Realtime facial animation with on-the-fly correctives

Abstract

Figure 1: Our adaptive tracking model conforms to the input expressions on-the-fly, producing a better fit to the user than state-of-the-art data driven techniques [Weise et al. 2011] which are confined to learned motion priors and generate plausible but not accurate tracking. AbstractWe introduce a real-time and calibration-free facial performance capture framework based on a sensor with video and depth input. In this framework, we develop an adaptive PCA model using shape correctives that adjust on-the-fly to the actor's expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, we do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, we use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, we use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. We show that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. We demonstrate that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character.

Unconstrained realtime facial performance capture

“…We detect dynamic occlusions caused by temporal shape and texture variations using an outlier voting scheme in superpixel space. As recently demonstrated by Li et al [31], the combination of sparse 2D facial features (e.g., eyes, eyebrows, and mouth) with dense depth maps are particularly effective in improving tracking fidelity. However, because facial landmark detection becomes significantly less reliable when the face is occluded, we synthesize plausible face textures right after our face segmentation step.…”

“…The data-driven method of Weise et al [39] uses a motion prior database to handle noise and the lowresolution depth maps from the Kinect sensor. For improved fidelity, techniques that combine depth input data with sparse facial features were introduced [31,11,13,9]. To improve accessibility with less input training, an example-based facial rigging method was introduced by Li et al [30].…”

“…To improve accessibility with less input training, an example-based facial rigging method was introduced by Li et al [30]. The method of [31] builds a single neutral model before tracking and trains PCA-based correctives for the expressions during tracking with samples obtained from per-vertex Laplacian deformations. Bouaziz et al [6] introduce a completely calibration-free system by modeling the full blendshape model during tracking.…”

Real-Time Enhancement of Dynamic Depth Videos with Non-Rigid Deformations

“…These techniques, however, cannot handle large deformations, and are not very practical for real-time applications. Real-time non-rigid reconstruction approaches have been achieved with the help of a template which is first acquired then used for tracking of non-rigidities with a good flexibility [40], [41]. Recently, we have proposed KinectDeform [11], the first nonrigid version of KinectFusion.…”

DART: Dense Articulated Real-Time Tracking

“…The application-specific nature of these approaches enables their authors to show excellent performance by taking advantage of domain-specific features and constraints, but it also prevents them from serving as general tools for tracking arbitrary articulated objects. Techniques have also been developed to track fully non-rigid deformations of an underlying surface template for both specific [21] and general [14,20,28] object cases. However, the full generality of these models comes at the cost of increased model complexity, and for many objects that are well modelled as piecewise rigid bodies, such overparameterized output obscures the utility of tracking the articulated body state directly.…”