Unsupervised Part-Based Disentangling of Object Shape and Appearance

Abstract

Large intra-class variation is the result of changes in multiple object characteristics. Images, however, only show the superposition of different variable factors such as appearance or shape. Therefore, learning to disentangle and represent these different characteristics poses a great challenge, especially in the unsupervised case. Moreover, large object articulation calls for a flexible part-based model. We present an unsupervised approach for disentangling appearance and shape by learning parts consistently over all instances of a category. Our model for learning an object representation is trained by simultaneously exploiting invariance and equivariance constraints between synthetically transformed images. Since no part annotation or prior information on an object class is required, the approach is applicable to arbitrary classes. We evaluate our approach on a wide range of object categories and diverse tasks including pose prediction, disentangled image synthesis, and video-to-video translation. The approach outperforms the state-of-the-art on unsupervised keypoint prediction and compares favorably even against supervised approaches on the task of shape and appearance transfer.

Self-Supervised Learning of Interpretable Keypoints From Unlabelled Videos

“…Jakab & Gupta et al [25], the most related, is described in the introduction. Lorenz et al [33], Zhang et al [74] develop an auto-encoding formulation to discover landmarks as explicit structural representations for a given…”

“…Our method allows for similar but more fine-grained conditional image generation, conditioned on an appearance image or object landmarks. Many unsupervised methods for pose estimation [25,33,50,67,74] share similar ability. However, we can achieve more accurate and predictable image editing by manipulating semantic parts in the image through their corresponding landmarks.…”

“…Our method obtains state-of-the-art landmark detection performance for approaches that use unlabelled images for supervision. In contrast, self-supervised landmark detectors [25,33,54,74] can only learn to discover keypoints [right] that are not human-interpretable (predictions from [25]) and require supervised post-processing.…”

360-Degree Textures of People in Clothing from a Single Image

“…Pose guided image and video generation Given a source image and a target 2D pose, image-based methods [43,44,22,59,49] produce an image with the source appearance in the target pose. To deal with pixel miss-alignments, it is helpful to transform the pixels in the original image to match the target pose within the network [60,11,42]. Following similar ideas to growing GANs [36], high resolution anime characters can be generated [29].…”

Unsupervised Robust Disentangling of Latent Characteristics for Image Synthesis

“…[17] proposes a cycle-consistent VAE which adds a cyclic loss to the VAE objective. [40] directly models π as keypoints. All of these methods rely on the same basic principle for disentanglement: Constraining the amount of information in π.…”