Real-time expression transfer for facial reenactment

Abstract

We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor. The novelty of our approach lies in the transfer and photorealistic re-rendering of facial deformations and detail into the target video in a way that the newly-synthesized expressions are virtually indistinguishable from a real video. To achieve this, we accurately capture the facial performances of the source and target subjects in real-time using a commodity RGB-D sensor. For each frame, we jointly fit a parametric model for identity, expression, and skin reflectance to the input color and depth data, and also reconstruct the scene lighting. For expression transfer, we compute the difference between the source and target expressions in parameter space, and modify the target parameters to match the source expressions. A major challenge is the convincing re-rendering of the synthesized target face into the corresponding video stream. This requires a careful consideration of the lighting and shading design, which both must correspond to the real-world environment. We demonstrate our method in a live setup, where we modify a video conference feed such that the facial expressions of a different person (e.g., translator) are matched in real-time.

Reconstruction of Personalized 3D Face Rigs from Monocular Video

“…Our sparse regression takes 10ms for the medium layer and 2sec for the fine-scale detail layer. We believe that a drastic reduction of the computation time is possible by harnessing the data-parallel processing power of modern GPUs, as recently demonstrated for nonlinear optimization [Thies et al 2015;]. …”

“…Shi et al [2014] use a very similar tracking approach, but do not extract a high-fidelity parametrized 3D rig that contains a generative wrinkle formation model capturing the person-specific idiosyncrasies. Recently, Thies et al [2015] presented an approach for real-time facial reenactment, but the method cannot handle fine-scale surface detail and requires RGB-D camera input. Cao et al [2014a] use a learned regression model to fit, in real time, a generic identity and expression model to RGB face video.…”

Realtime Dynamic 3D Facial Reconstruction for Monocular Video In-the-Wild

“…A depth based method was introduced, which uses a parametric 3D facial model to robustly deal with the noisy depth input in [37]. Recently a state-of-theart depth based tracking method with parametric 3D facial geometry and lighting model has been proposed for realtime facial expression transfer and reenactment in [33]. Due to the limited depth sensor resolution, RGB color input is used to supplement extra information to refine the tracking.…”

Real‐time facial expression transfer with single video camera

“…Our method instead deals with transferring expression onto a target character from images or videos, where previous approaches are either not applicable or rely on depth information . We need to track the facial mesh of the actor and target as well as modelling their appearances, which is difficult to achieve real‐time performance.…”