GANimation: One-Shot Anatomically Consistent Facial Animation

Abstract

Recent advances in Generative Adversarial Networks (GANs) have shown impressive results for the task of facial expression synthesis. The most successful architecture is StarGAN [5], that conditions GANs' generation process with images of a specific domain, namely a set of images of people sharing the same expression. While effective, this approach can only generate a discrete number of expressions, determined by the content and granularity of the dataset. To address this limitation, in this paper, we introduce a novel GAN conditioning scheme based on Action Units (AU) annotations, which describes in a continuous manifold the anatomical facial movements defining a human expression. Our approach allows controlling the magnitude of activation of each AU and combining several of them. Additionally, we propose a weakly supervised strategy to train the model, that only requires images annotated with their activated AUs, and exploit a novel self-learned attention mechanism that makes our network robust to changing backgrounds, lighting conditions and occlusions. Extensive evaluation shows that our approach goes beyond competing conditional generators both in the capability to synthesize a much wider range of expressions ruled by anatomically feasible muscle movements, as in the capacity of dealing with images in the wild.

Masked Linear Regression for Learning Local Receptive Fields for Facial Expression Synthesis

“…In contrast, our proposed method preserved the cat's original features (see third row of Figure 1). Figure 21 compares the proposed method with the expression transfer results of GANimation (Pumarola et al 2019). Fig.…”

“…This becomes more challenging if the trained models are to be deployed in resource-constrained environments such as mobile devices and embedded systems with limited memory, computational power, and stored energy. A comparison of the proposed algorithm with four state-of-theart GAN models including Pix2Pix (Isola et al 2017), Cy-cleGAN (Zhu et al 2017), StarGAN (Choi et al 2018) and GANimation (Pumarola et al 2019) is shown in Table 1. The proposed algorithm has more than two orders of magnitude fewer number of parameters than each of these GANs.…”

“…The proposed approach also exhibits an advantage over GAN models in terms of generalization. Figure 1 shows a comparison of happy expressions synthesized for pencil sketches and several animal faces by the proposed algorithm and by Pix2Pix (Isola et al 2017), CycleGAN (Zhu et al 2017), StarGAN (Choi et al 2018) and GANimation (Pumarola et al 2019). All methods were trained entirely on real human faces, therefore these test images may be considered as out-of-dataset.…”

Generative Video Face Reenactment by AUs and Gaze Regularization

“…Thanks to the advances in deep learning techniques and appearance of big datasets, quite realistic images can be generated by these approaches. The majority of these approaches are encoder-decoder networks that receive an input image and are conditioned to the modification parameters, such as facial expressions [13]. An example of generative face reenactment is the work of [5], where an avatar human agent dynamically interacts with users.…”

Special Issue: Advances in Architectures and Theories for Computer Vision

“…Often the contribution is a new architecture that is better suited to a particular vision task (e.g. Law and Deng 2019, Veit and Belongie 2019, Esteves et al 2019 or the definition of a new visual task that can be tackled with this powerful tool (Harwath et al 2019, Pumarola et al 2019. The flow of ideas between Computer Vision and Deep Learning continues to be bidirectional, and one of the selected papers defines a new learning procedure that may prove to be useful beyond computer vision (Wu and He 2019).…”