Deep video portraits

Abstract

We present a novel approach that enables photo-realistic re-animation of portrait videos using only an input video. In contrast to existing approaches that are restricted to manipulations of facial expressions only, we are the first to transfer the full 3D head position, head rotation, face expression, eye gaze, and eye blinking from a source actor to a portrait video of a target actor. The core of our approach is a generative neural network with a novel space-time architecture. The network takes as input synthetic renderings of a parametric face model, based on which it predicts photo-realistic video frames for a given target actor. The realism in this rendering-to-video transfer is achieved by careful adversarial training, and as a result, we can create modified target videos that mimic the behavior of the synthetically-created input. In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network - thus taking full control of the target. With the ability to freely recombine source and target parameters, we are able to demonstrate a large variety of video rewrite applications without explicitly modeling hair, body or background. For instance, we can reenact the full head using interactive user-controlled editing, and realize high-fidelity visual dubbing. To demonstrate the high quality of our output, we conduct an extensive series of experiments and evaluations, where for instance a user study shows that our video edits are hard to detect.

Neural Rerendering in the Wild

“…Image translation techniques can be used to rerender scenes in a more realistic domain, to enable facial expression synthesis [20], to fix artifacts in captured 3D performances [28], or to add viewpoint-dependent effects [44]. In our paper, we demonstrate an approach for training a neural rerendering framework in the wild, i.e., with uncontrolled data instead of captures under constant lighting conditions.…”

“…We adapt recent neural rerendering frameworks [20,28] to work with unstructured photo collections. Given a large internet photo collection {I i } of a scene, we first generate a proxy 3D reconstruction using COLMAP [36,37,38], which applies Structure-from-Motion (SfM) and Multi-View Stereo (MVS) to create a dense colored point cloud.…”

“…However, the image-to-image translation paradigm used in [20,28] is not appropriate for our use case, as it assumes a one-to-one mapping between inputs and outputs. A scene observed from a particular viewpoint can look very different depending on weather, lighting conditions, color balance, post processing filters, etc.…”

Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos

“…The requirements for manipulating or synthesizing videos were dramatically simplified when it became possible to create forged videos from only a short video of the target person [5,7] and then from a single ID photo [8] following the acting of an actor. Suwajanakorn et al's mapping method [9] enhanced the ability of manipulators to learn the mapping between speech and lip motion.…”

Neural Re-Simulation for Generating Bounces in Single Images

“…In particular, here we apply neural re-simulation from trajectories with noisy and insufficient data to plausible output. Our solution is thus also related to recent approaches for re-rendering scenes with a neural network [21,27,29,44]; here we seek to re-simulate dynamic trajectory outputs.…”