3D Guided Fine-Grained Face Manipulation

Geng, Zhenglin; Cao, Chen; Tulyakov, Sergey

doi:10.1109/cvpr.2019.01005

Cited by 90 publications

(67 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These approaches use additional information such as conditioning labels (e.g. indicating a facial expression, a body pose) [49,35,16,39]. More specifically, they are purely data-driven, leveraging a large collection of training data to learn a latent representation of the visual inputs for synthesis.…”

Section: Introductionmentioning

confidence: 99%

Animating Arbitrary Objects via Deep Motion Transfer

Siarohin

Lathuilière

Tulyakov

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

351

379

View full text Add to dashboard Cite

This paper introduces a novel deep learning framework for image animation. Given an input image with a target object and a driving video sequence depicting a moving object, our framework generates a video in which the target object is animated according to the driving sequence. This is achieved through a deep architecture that decouples appearance and motion information. Our framework consists of three main modules: (i) a Keypoint Detector unsupervisely trained to extract object keypoints, (ii) a Dense Motion prediction network for generating dense heatmaps from sparse keypoints, in order to better encode motion information and (iii) a Motion Transfer Network, which uses the motion heatmaps and appearance information extracted from the input image to synthesize the output frames. We demonstrate the effectiveness of our method on several benchmark datasets, spanning a wide variety of object appearances, and show that our approach outperforms stateof-the-art image animation and video generation methods. Our source code is publicly available 1 .

show abstract

Section: Introductionmentioning

confidence: 99%

Animating Arbitrary Objects via Deep Motion Transfer

Siarohin

Lathuilière

Tulyakov

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

351

379

View full text Add to dashboard Cite

show abstract

“…Finally, we acknowledge (Geng et al 2019) which is a concurrent work, very closely related to ours. In this work the authors similarly to us employ blendshape parameters for expression editing but follow a different approach in image editing, handling 3D texture (UV maps) and shape separately and composing them in a final output image by rendering.…”

Section: Facial Attribute Editing and Reenactment In Imagesmentioning

confidence: 78%

SliderGAN: Synthesizing Expressive Face Images by Sliding 3D Blendshape Parameters

Ververas

Zafeiriou

2020

Int J Comput Vis

View full text Add to dashboard Cite

Image-to-image (i2i) translation is the dense regression problem of learning how to transform an input image into an output using aligned image pairs. Remarkable progress has been made in i2i translation with the advent of deep convolutional neural networks and particular using the learning paradigm of generative adversarial networks (GANs). In the absence of paired images, i2i translation is tackled with one or multiple domain transformations (i.e., CycleGAN, StarGAN etc.). In this paper, we study the problem of image-to-image translation, under a set of continuous parameters that correspond to a model describing a physical process. In particular, we propose the SliderGAN which transforms an input face image into a new one according to the continuous values of a statistical blendshape model of facial motion. We show that it is possible to edit a facial image according to expression and speech blendshapes, using sliders that control the continuous values of the blendshape model. This provides much more flexibility in various tasks, including but not limited to face editing, expression transfer and face neutralisation, comparing to models based on discrete expressions or action units.

show abstract

“…They use two priors: SMPL [32] parametric human mesh model, and a prior on 3D poses acquired via adversarial learning from mocap data. Analogous works include [16,17,18,46,49,61,65,69].…”

Section: Related Workmentioning

confidence: 99%

Self-Supervised Learning of Interpretable Keypoints From Unlabelled Videos

Jakab

Gupta

Bilen

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We propose a new method for recognizing the pose of objects from a single image that for learning uses only unlabelled videos and a weak empirical prior on the object poses. Video frames differ primarily in the pose of the objects they contain, so our method distils the pose information by analyzing the differences between frames. The distillation uses a new dual representation of the geometry of objects as a set of 2D keypoints, and as a pictorial representation, i.e. a skeleton image. This has three benefits: (1) it provides a tight 'geometric bottleneck' which disentangles pose from appearance, (2) it can leverage powerful image-to-image translation networks to map between photometry and geometry, and (3) it allows to incorporate empirical pose priors in the learning process. The pose priors are obtained from unpaired data, such as from a different dataset or modality such as mocap, such that no annotated image is ever used in learning the pose recognition network. In standard benchmarks for pose recognition for humans and faces, our method achieves state-of-the-art performance among methods that do not require any labelled images for training

show abstract

3D Guided Fine-Grained Face Manipulation

Cited by 90 publications

References 32 publications

Animating Arbitrary Objects via Deep Motion Transfer

Animating Arbitrary Objects via Deep Motion Transfer

SliderGAN: Synthesizing Expressive Face Images by Sliding 3D Blendshape Parameters

Self-Supervised Learning of Interpretable Keypoints From Unlabelled Videos

Contact Info

Product

Resources

About