Depth-Aware Generative Adversarial Network for Talking Head Video Generation

Abstract

Predominant techniques on talking head generation largely depend on 2D information, including facial appearances and motions from input face images. Nevertheless, dense 3D facial geometry, such as pixel-wise depth, plays a critical role in constructing accurate 3D facial structures and suppressing complex background noises for generation. However, dense 3D annotations for facial videos is prohibitively costly to obtain. In this work, firstly, we present a novel self-supervised method for learning dense 3D facial geometry (i.e., depth) from face videos, without requiring camera parameters and 3D geometry annotations in training. We further propose a strategy to learn pixel-level uncertainties to perceive more reliable rigid-motion pixels for geometry learning. Secondly, we design an effective geometry-guided facial keypoint estimation module, providing accurate keypoints for generating motion fields. Lastly, we develop a 3D-aware cross-modal (i.e., appearance and depth) attention mechanism, which can be applied to each generation layer, to capture facial geometries in a coarse-to-fine manner. Extensive experiments are conducted on three challenging benchmarks (i.e., VoxCeleb1, VoxCeleb2, and HDTF). The results demonstrate that our proposed framework can generate highly realistic-looking reenacted talking videos, with new state-of-the-art performances established on these benchmarks. The codes and trained models are publicly available on the GitHub project page.

Audio-Visual Face Reenactment

“…Talking head generation works can be broadly classified in three categories based on the type of input they use to generate a talking head: Text-driven [16,33,36], Audio-driven [9,13,18,31,37,43,45], and Video-driven [12,27,29,39,44] Talking Head Generation.…”

“…The motion field was used to calculate dense flow and warp the source frame in a latent space. Several other works [39,12] followed the same principle and added supplementary components to improve the quality. Face-vid2vid [39] used keypoint information in a 3D space, taking care of head rotation, among other things.…”

“…Face-vid2vid [39] used keypoint information in a 3D space, taking care of head rotation, among other things. DA-GAN [12] further added depth-aware attention to provide dense 3D facial geometry to guide the generation of motion fields. A similar approach in Motion-Representation-in-Articulated-Animation [30] uses key regions instead of keypoints to generate the warpable motion field.…”

“…Multiple publications have improved the quality of the generations. Existing works on talking head generation generally use a single modality, i.e., either visual [12,29,39,40] or audio features [13,37,31]. Audio-driven talking head generation models are good at generating quality lipsync; however, they have a serious drawback in handling non-verbal cues.…”

“…The video-driven methods heavily rely on the disentanglement of motion from the appearance [17]. These methods generally use key points as an intermediate representation [29,12,39] and try to align the detected key points of source and driving frames. These works learn key points in an unsupervised manner and fail to focus on specific regions of the face.…”