Unsupervised Robust Disentangling of Latent Characteristics for Image Synthesis

Esser, Patrick; Haux, Johannes; Ommer, Björn

doi:10.1109/iccv.2019.00279

Cited by 38 publications

(23 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Siarohin et al learn unsupervised keypoints in order to warp object parts into novel poses [56,57]. Lorenz et al [37] show that body parts can also be represented by unsupervised learning, which also helps to disentangle body pose and shape [11]. Nevertheless, without the benefit of explicit pose information from a human detection model, these methods struggle to generate good results for challenging driving poses.…”

Section: Related Workmentioning

confidence: 99%

Flow Guided Transformable Bottleneck Networks for Motion Retargeting

Ren

Chai

Woodford

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Human motion retargeting aims to transfer the motion of one person in a "driving" video or set of images to another person. Existing efforts leverage a long training video from each target person to train a subject-specific motion transfer model. However, the scalability of such methods is limited, as each model can only generate videos for the given target subject, and such training videos are labor-intensive to acquire and process. Few-shot motion transfer techniques, which only require one or a few images from a target, have recently drawn considerable attention. Methods addressing this task generally use either 2D or explicit 3D representations to transfer motion, and in doing so, sacrifice either accurate geometric modeling or the flexibility of an end-to-end learned representation. Inspired by the Transformable Bottleneck Network, which renders novel views and manipulations of rigid objects, we propose an approach based on an implicit volumetric representation of the image content, which can then be spatially manipulated using volumetric flow fields. We address the challenging question of how to aggregate information across different body poses, learning flow fields that allow for combining content from the appropriate regions of input images of highly non-rigid human subjects performing complex motions into a single implicit volumetric representation. This allows us to learn our 3D representation solely from videos of moving people. Armed with both 3D object understanding and end-to-end learned rendering, this categorically novel representation delivers state-of-the-art image generation quality, as shown by our quantitative and qualitative evaluations.

show abstract

Section: Related Workmentioning

confidence: 99%

Flow Guided Transformable Bottleneck Networks for Motion Retargeting

Ren

Chai

Woodford

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…Image methods. Controllable generative models have also been developed for images (Härkönen et al, 2020;Esser et al, 2019;Singh et al, 2019;Lample et al, 2017;Karras et al, 2020;Brock et al, 2019;Collins et al, 2020;Shen et al, 2020;Esser et al, 2020;Goetschalckx et al, 2019;Pavllo et al, 2020;, which control the object class, pose, lighting, etc., of an image. Many image style transform methods have also been developed Gatys et al, 2016).…”

Section: Related Workmentioning

confidence: 99%

Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models

Chang¹,

Shrivastava²,

Koppula³

et al. 2021

Preprint

View full text Add to dashboard Cite

Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, typical training algorithms for these controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but different samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. By introducing a style transformation module that we call style equalization, we enable training using different content and style samples and thereby mitigate the traininginference mismatch. To demonstrate its generality, we applied style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. Our models achieve state-of-the-art style replication with a similar mean style opinion score as the real data. Moreover, the proposed method enables style interpolation between sequences and generates novel styles. 1 * work done during internship at Apple 1 Our synthesis results can be viewed at https://apple.github.io/ml-style-equalization

show abstract

“…Other works. Esser et al adopt the disentanglement of the human body pose from the corresponding appearance (style) information in the context of a dual-encoder VAE setting, however they use the body-related factors for other human appearance transfer [75] and synthesis [108].…”

Section: Tips and Tricksmentioning

confidence: 99%

Learning Disentangled Representations in the Imaging Domain

Liu,

Sanchez,

Thermos

et al. 2021

Preprint

View full text Add to dashboard Cite

Disentangled representation learning has been proposed as an approach to learning general representations. This can be done in the absence of, or with limited, annotations. A good general representation can be readily fine-tuned for new target tasks using modest amounts of data, or even be used directly in unseen domains achieving remarkable performance in the corresponding task. This alleviation of the data and annotation requirements offers tantalising prospects for tractable and affordable applications in computer vision and healthcare. Finally, disentangled representations can offer model explainability and can help us understand the underlying causal relations of the factors of variation, increasing their suitability for real-world deployment. In this tutorial paper, we will offer an overview of the disentangled representation learning, its building blocks and criteria, and discuss applications in computer vision and medical imaging. We conclude our tutorial by presenting the identified opportunities for the integration of recent machine learning advances into disentanglement, as well as the remaining challenges.

show abstract

Unsupervised Robust Disentangling of Latent Characteristics for Image Synthesis

Cited by 38 publications

References 37 publications

Flow Guided Transformable Bottleneck Networks for Motion Retargeting

Flow Guided Transformable Bottleneck Networks for Motion Retargeting

Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models

Learning Disentangled Representations in the Imaging Domain

Contact Info

Product

Resources

About