Realistic Face Reenactment via Self-Supervised Disentangling of Identity and Pose

Abstract

Recent works have shown how realistic talking face images can be obtained under the supervision of geometry guidance, e.g., facial landmark or boundary. To alleviate the demand for manual annotations, in this paper, we propose a novel self-supervised hybrid model (DAE-GAN) that learns how to reenact face naturally given large amounts of unlabeled videos. Our approach combines two deforming autoencoders with the latest advances in the conditional generation. On the one hand, we adopt the deforming autoencoder to disentangle identity and pose representations. A strong prior in talking face videos is that each frame can be encoded as two parts: one for video-specific identity and the other for various poses. Inspired by that, we utilize a multi-frame deforming autoencoder to learn a pose-invariant embedded face for each video. Meanwhile, a multi-scale deforming autoencoder is proposed to extract pose-related information for each frame. On the other hand, the conditional generator allows for enhancing fine details and overall reality. It leverages the disentangled features to generate photo-realistic and pose-alike face images. We evaluate our model on VoxCeleb1 and RaFD dataset. Experiment results demonstrate the superior quality of reenacted images and the flexibility of transferring facial movements between identities.

A comprehensive survey on semantic facial attribute editing using generative adversarial networks

“…The model, named ICface, is based on two generators: one for generating a neutral image from the target person and one for manipulating it as per the extracted non-identity attributes of the input face. As another example of these models, DAE-GAN [200] learns separate pose and identity representations for generating talking faces. A conditional GAN is then employed to perform the style transformation.…”

Region-Aware Face Swapping

“…Feature fusion is an important process in face swapping. Most previous works [12,28,29,47,50,53] are inspired by style transfer methods. They employ AdaIN [21] to inject the identity vector into the target face to generate swapped face.…”

Preserving Integrity in Online Social Networks

“…With the recent progress in computer vision and speech processing and the advent of generative adversarial networks, manipulated media has leaped a significant step forward. These advances have unleashed potential for harmful applications like face puppeteering [83,136], speech manipulation [97], face transfer [115], full body manipulation [23], and other more general media manipulations. Many of these manipulations can be used to impersonate others, spread false information, or just introduce bias in the observer.…”