Face transfer with multilinear models

Abstract

Figure 1: Face Transfer with multilinear models gives animators decoupled control over facial attributes such as identity, expression, and viseme. In this example, we combine pose and identity from the first frame, surprised expression from the second, and a viseme (mouth articulation for a sound midway between "oo" and "ee") from the third. The resulting composite is blended back into the original frame. AbstractFace Transfer is a method for mapping videorecorded performances of one individual to facial animations of another. It extracts visemes (speech-related mouth articulations), expressions, and three-dimensional (3D) pose from monocular video or film footage. These parameters are then used to generate and drive a detailed 3D textured face mesh for a target identity, which can be seamlessly rendered back into target footage. The underlying face model automatically adjusts for how the target performs facial expressions and visemes. The performance data can be easily edited to change the visemes, expressions, pose, or even the identity of the target-the attributes are separably controllable. This supports a wide variety of video rewrite and puppetry applications.Face Transfer is based on a multilinear model of 3D face meshes that separably parameterizes the space of geometric variations due to different attributes (e.g., identity, expression, and viseme). Separability means that each of these attributes can be independently varied. A multilinear model can be estimated from a Cartesian product of examples (identities × expressions × visemes) with techniques from statistical analysis, but only after careful preprocessing of the geometric data set to secure one-to-one correspondence, to minimize cross-coupling artifacts, and to fill in any missing examples. Face Transfer offers new solutions to these problems and links the estimated model with a face-tracking algorithm to extract pose, expression, and viseme parameters.

Inverse Rendering of Faces with a 3D Morphable Model

“…When data is missing, the gaps have to be filled. A demonstration of video driven animation with multilinear models (including the missing data problem), was demonstrated by Vlasic et al [24]. Although an experienced observer can make out the manipulation, the approach looks very promising.…”

“…Compared to matrix SVD, N-mode SVD, does not result in an optimal solution and further refinement is required [24]. Unfortunately, many matrix SVD properties do not hold for N-mode SVD.…”

Bilinear Active Appearance Models

“…Multilinear face models have also been successfully applied by Vlasic et. all [41] to face analysis. However, they use 3D scanned face data and pose variations are not explictly modeled.…”

Vision Based Speech Animation Transferring with Underlying Anatomical Structure

“…The data analysis may be based on machine learning [1,2,4,6,7] or probabilistic framework [5]. Ezzat et al [1] employ a variant of MMM to synthesize mouth configurations of a novel speech.…”

“…Voice Puppetry [5] utilizes a probabilistic framework to find an optimal trajectory for the whole utterance based on facial gestures learned from videos. Vlasic et al [7] use a multilinear model to separate different attributes of facial models, e.g. expression and visemes, and connect multilinear model directly to videos for a time-series of poses and attribute parameters.…”