Modeling and animation of individualized faces for 3D facial expression synthesis

Abstract

This article presents new methods for realistic and fast 3D facial expression synthesis based on an individualized face model constructed from the anatomical perspective. The face model has a 3D structure of anatomically motivated facial skin, muscles, and skull. We start from a highly accurate facial mesh reconstructed from the individual face measurements. By taking as input a collection of reflectance images covering the face, the view-based texture blending method automatically generates a comprehensive texture map for photorealistic rendering. Based on the reconstructed facial mesh, a deformable multilayer skin model is developed to simulate the dynamic behavior of the skin by taking into account the nonlinear stress-strain relationship and incompressibility of the skin. We develop a muscle modeling process that includes three kinds of forcebased facial muscle model to simulate facial muscle contraction. The 3D face model incorporates a skull structure which extends the scope of facial motion, facilitates facial muscle construction and constrains skin deformation. Based on the facial action coding system, various facial expressions are synthesized by the contraction of a set of facial muscles. Lagrangian mechanics governs the dynamics, dictating the deformation of facial skin in response to the contraction of muscles. For computational efficiency, we devise an adaptive computation algorithm for the numerical simulation. It uses either a semi-implicit integration scheme or a quasi-static solver to compute the relaxation of the facial mesh by traversing the designed data structures in a breadth-first order. The resulting system enables us to construct realistic face models of living humans and to synthesize facial expressions in real time.

The 4D Space-Time Dimensions of Facial Perception

“…Facial movement can be modeled as rigid and non-rigid motion (for review see Smeets et al, 2012;Chrysos et al, 2018). Evidence suggests anatomically accurate rigid and non-rigid facial movements are essential to facial perception (Trautmann et al, 2009;Johnston et al, 2013) and is achievable computationally (Zhang et al, 2003;Zhang and Fisher, 2019). Deformable facial modeling, relates to movement we produce through voluntary, striated musculature or facial muscles.…”

“…From cognitive information such as identity and recognition; to detecting complex patterns of speech or emotion; to gleaning important social cues such as gender, ethnicity, age or health. The human face has been defined by other authors as a multi-dimensional pattern which evolves over time ( Zhang et al, 2003 ; Liu et al, 2012 ; Marcolin and Vezzetti, 2017 ). This multi-dimensional, complex pattern is also highly individualized; varying from person to person.…”

“…Three-dimensional objects or models rely upon a geometric structural and surface model presenting accurate reconstruction of breadth, height and importantly depth , while still being presented on a two-dimensional screen ( Hallinan et al, 1999 ; Marschner et al, 2000 ; Egger et al, 2019 ). Representing detailed facial tissue, bone structure and the musculature system, which make up the human face, has not been an easy feat, yet is now computationally achievable ( Zhang et al, 2003 ; Ekrami et al, 2018 ). Moreover, these 3D facial models are revealing key differences about both individual recognition, as well as categorization or classification of human faces as objects.…”

3D‐Facial Expression Synthesis and its Application to Face Recognition Systems

“…These early models oversimplified the biomechanics of the face by considering an infinitesimal skin surface with no underlying structure. Newer schemes, on the other hand, are getting closer to the modeling of every single layer of the facial structure, such as epidermis, dermis, hypodermis, fatty tissue, muscles, skull and movable jaws [36]. Terzopoulus et.al.…”