Head2Head++: Deep Facial Attributes Re-Targeting

Abstract

This document is made available in accordance with publisher policies and may differ from the published version or from the version of record. If you wish to cite this item you are advised to consult the publisher's version. Please see the URL above for details on accessing the published version.

Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos

“…Traditional techniques [42], [43] perform 3D face reconstruction on the reference video and render the target subject under the source expressions on top of the original target footage. Learning-based methods, like DVP [23] and Head2Head++ [13] use conditional GANs to render the target subject under the given conditions (expressions, pose, eye-gaze). Nevertheless, these methods offer no semantic control over the generated video, as they directly copy the expressions from a source actor.…”

“…We use FAN [4] to obtain 68 facial landmarks for each frame. Afterwards, similarly to [13], we estimate eye pupil coordinates based on the inverse intensities of the pixels within the eye area and create eye images E ∈ R 256×256×3 that provide the face renderer with information about the eyegaze. However, in contrast to [13], we only draw two red disks around eye pupils and not the edges of the outline.…”

“…3D Face Synthesis & Rendering: Having modified the expression parameters through our 3D-based Emotion Manipulator, we synthesize a manipulated 3D face geometry under the new emotion. We then render it (using conventional 3D graphics) to a convenient representation for neural rendering, the so-called Normalized Mean Face Coordinate (NMFC) image [13], and concatenate it with the similarlyrendered detailed shape image S and eye image E. Neural Face Renderer: We feed our neural renderer with the NMFC, S and E images as conditional input. We build it upon the publicly available Head2Head++ [13] implementation and train it on the training footage of a target actor in a self-reenactment fashion (i.e.…”

“…More related to this task, is the face reenactment problem, where the facial performance of a source actor is transferred to a target one, making the latter mimic the expressions of the former. State-of-the-art techniques [13,23,53] achieve compelling photo-realism by training a neural renderer conditioned on a facial representation (e.g. 3DMMs).…”

HeadGAN: One-shot Neural Head Synthesis and Editing

“…For instance, Face2Face [40] method performs face reenactment, by recovering facial expressions from a driving video and overwriting them to the source frames. Some recent learningbased approaches [23,25,13] have sought to solve the problem of full head reenactment, which aims to transfer not only the expression, but also the pose, from a driving person to the source identity. The shortcoming of such methods is their dependence on long video footage of the source, as they train person-specific models.…”