Learning a model of facial shape and expression from 4D scans

Abstract

The field of 3D face modeling has a large gap between high-end and low-end methods. At the high end, the best facial animation is indistinguishable from real humans, but this comes at the cost of extensive manual labor. At the low end, face capture from consumer depth sensors relies on 3D face models that are not expressive enough to capture the variability in natural facial shape and expression. We seek a middle ground by learning a facial model from thousands of accurately aligned 3D scans. Our FLAME model (Faces Learned with an Articulated Model and Expressions) is designed to work with existing graphics software and be easy to fit to data. FLAME uses a linear shape space trained from 3800 scans of human heads. FLAME combines this linear shape space with an articulated jaw, neck, and eyeballs, pose-dependent corrective blendshapes, and additional global expression blendshapes. The pose and expression dependent articulations are learned from 4D face sequences in the D3DFACS dataset along with additional 4D sequences. We accurately register a template mesh to the scan sequences and make the D3DFACS registrations available for research purposes. In total the model is trained from over 33, 000 scans. FLAME is low-dimensional but more expressive than the FaceWarehouse model and the Basel Face Model. We compare FLAME to these models by fitting them to static 3D scans and 4D sequences using the same optimization method. FLAME is significantly more accurate and is available for research purposes (http://flame.is.tue.mpg.de).

Multilinear Autoencoder for 3D Face Model Learning

“…Li et al [18] recently introduced a generative model learned from a large collection of 3D motion sequences of faces. Pose changes due to skeletal motion is modeled using a skinning approach, while shape changes due to identity, expression, and pose correction are modeled as linear factors similar to 3DMM.…”

“…The input scans are registered using a non-rigid approach, e.g. [2,18], which enables reconstruction errors between the output meshes and the input scans to be estimated in a consistent way. These registrations need not be precise since they only serve as initialization and will be refined.…”

“…We will refer to this MAE as Bu+Bosph. The second MAE is learned by combining the previous with a subset of the dynamic database D3DFACS [8] using the publicly available registrations of Li et al [18], which allows to build a considerably larger training set. We will refer to this MAE as D3DFACS.…”

“…This is especially true with the recent increase of available databases of 3D face scans that can be used as training data, e.g. [18].…”

“…Acknowledgements We would like to thank Timo Bolkart for providing the registrations for D3DFACS data [18], and Mengjiao Wang for providing the code of [27].…”