Facial performance synthesis using deformation-driven polynomial displacement maps

Abstract

Motion captureDeformed Adding medium Adding high Ground truth markers neutral mesh frequency displacements frequency displacements geometry Figure 1: We synthesize new high-resolution geometry and surface detail from sparse motion capture markers using deformation-driven polynomial displacement maps; our results agree well with high-resolution ground truth geometry of dynamic facial performances. AbstractWe present a novel method for acquisition, modeling, compression, and synthesis of realistic facial deformations using polynomial displacement maps. Our method consists of an analysis phase where the relationship between motion capture markers and detailed facial geometry is inferred, and a synthesis phase where novel detailed animated facial geometry is driven solely by a sparse set of motion capture markers. For analysis, we record the actor wearing facial markers while performing a set of training expression clips. We capture real-time high-resolution facial deformations, including dynamic wrinkle and pore detail, using interleaved structured light 3D scanning and photometric stereo. Next, we compute displacements between a neutral mesh driven by the motion capture markers and the high-resolution captured expressions. These geometric displacements are stored in a polynomial displacement map which is parameterized according to the local deformations of the motion capture dots. For synthesis, we drive the polynomial displacement map with new motion capture data. This allows the recreation of large-scale muscle deformation, medium and fine wrinkles, and dynamic skin pore detail. Applications include the compression of existing performance data and the synthesis of new performances. Our technique is independent of the underlying geometry capture system and can be used to automatically generate high-frequency wrinkle and pore details on top of many existing facial animation systems.

Automatic acquisition of high-fidelity facial performances using monocular videos

“…One solution to the problem of marker-less facial capture is the use of depth and/or color data obtained from structured light systems [Zhang et al 2004;Ma et al 2008;Li et al 2009;Weise et al 2009]. For example, Zhang and colleagues [2004] captured 3D facial geometry and texture over time and built the correspondences across all the facial geometries by deforming a generic face template to fit the acquired depth data using optical flow computed from image sequences.…”

“…For example, Zhang and colleagues [2004] captured 3D facial geometry and texture over time and built the correspondences across all the facial geometries by deforming a generic face template to fit the acquired depth data using optical flow computed from image sequences. Ma et al [2008] achieved high-resolution facial reconstructions by interleaving structured light with spherical gradient photometric stereo using the USC Light Stage. Recently, Li and his colleagues [2009] captured dynamic depth maps with their realtime structured light system and fitted a smooth template to the captured depth maps.…”

Dynamic 3D avatar creation from hand-held video input

“…The work of Alexander et al [2013] recently extended this approach to enable real-time rendering of highly detailed facial rigs. Structured light and laser scanners have also been used to acquire facial geometry at the wrinkle scale [Zhang et al 2004;Ma et al 2008;Li et al 2009;Huang et al 2011]. Similarly, the setup of [Beeler et al 2010;Beeler et al 2011] is capable of reconstructing fine-scale detail using multiple calibrated/synchronized DSLR cameras.…”

A Practical Approach for Identity-Embodied 3D Artistic Face Modeling

“…Researchers have also developed a variety of algorithms, interfaces, and tools to allow users to directly manipulate 3D facial mesh models [22,23]. Recent data-driven facial editing and deformation algorithms utilize the statistical correlations in precollected facial datasets [24][25][26][27][28]. Notably, Blanz and Vetter [29] build a 3D morphable face model by constructing principal component analysis (PCA) spaces from the 3D geometry and texture of a scanned 3D face dataset and demonstrate that the constructed face PCA spaces can be used for various face synthesis and editing applications.…”