Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos

Abstract

Unsupervised landmark learning is the task of learning semantic keypoint-like representations without the use of expensive input keypoint annotations. A popular approach is to factorize an image into a pose and appearance data stream, then to reconstruct the image from the factorized components. The pose representation should capture a set of consistent and tightly localized landmarks in order to facilitate reconstruction of the input image. Ultimately, we wish for our learned landmarks to focus on the foreground object of interest. However, the reconstruction task of the entire image forces the model to allocate landmarks to model the background. Using a motion-based foreground assumption, this work explores the effects of factorizing the reconstruction task into separate foreground and background reconstructions in an unsupervised way, allowing the model to condition only the foreground reconstruction on the unsupervised landmarks. Our experiments demonstrate that the proposed factorization results in landmarks that are focused on the foreground object of interest when measured against ground-truth foreground masks. Furthermore, the rendered background quality is also improved as ill-suited landmarks are no longer forced to model this content. We demonstrate this improvement via improved image fidelity in a video-prediction task. Code is available at https://github.com/NVIDIA/UnsupervisedLandmarkLearning

Unsupervised View-Invariant Human Posture Representation

“…Unsupervised Pose Representation -There are several recent examples of RGB-based unsupervised learning approaches to 3D pose estimation, such as [23,24,25,26,27,28]. Authors in [24,25,26] extract unsupervised pose features from 2D joints generated from RGB data.…”

“…For example, Chen et al [24] train a network through a 2D-3D consistency loss, computed after lifting 2D pose to 3D joints and reprojecting 3D onto 2D. Dundar et al [28] disentangle pose and appearance features from an RGB image by designing a self-supervised auto-encoder that reconstructs an input image into foreground and background with the constraint that the appearance features remain consistent temporally while the pose features change. Honari et al [27] also relies on temporal information and factorizes the pose and appearance features in a contrastive learning manner.…”

“…Our key contributions can be summarized as follows: (i) we propose a novel unsupervised method that learns viewinvariant 3D pose representation from a 2D image without using 3D skeleton data and camera parameters. Our viewinvariant features can be applied directly by downstream tasks to be resilient to human pose variations in unseen viewpoints, unlike unsupervised 3D pose estimation methods such as [23,24,25,26,27,28] which obtain viewspecific 3D pose features, and require camera parameters and further steps to align their view-specific features in a canonical space, (ii) we introduce novel view-invariance and equivariance losses that impose on the network to preserve geometrical and positional order consistency of pose features -these losses can benefit the training process in other pretext tasks that exploit landmark representation, (iii) we evaluate the performance of learned pose features on two downstream tasks that demand view-invariancy and achieve state-of-the-art unsupervised cross-view action recognition accuracy on the NTU RGB+D standard benchmark dataset for RGB and depth images at 74.8% and 67.5% respectively, and for the first time we obtain unsupervised cross-view and cross-subject rank correlation results for human movement assessment scores on the QMAR dataset, while exceeding its supervised state-of-the-art results, (iv) we perform ablation studies to explore the impact of our loss functions on our proposed model.…”

View Generalization for Single Image Textured 3D Models

“…However, for this technology to be used in gaming, simulators, and virtual applications, they need to be fully controllable. Recent work controls image generation by conditioning output on different type of inputs such as natural and synthetic images [17,50,23,46,7,28], landmarks [27,39,8], and semantic maps [14,40,31]. Among these methods, [27,8,15] disentangle images into pose and appearance in an unsupervised way, and show control over pose during inference.…”

“…Recent work controls image generation by conditioning output on different type of inputs such as natural and synthetic images [17,50,23,46,7,28], landmarks [27,39,8], and semantic maps [14,40,31]. Among these methods, [27,8,15] disentangle images into pose and appearance in an unsupervised way, and show control over pose during inference. However, since they do not disentangle images into their 3D attributes, the control and manipulation over generated images are still limited.…”