Learning an animatable detailed 3D face model from in-the-wild images

Abstract

While current monocular 3D face reconstruction methods can recover fine geometric details, they suffer several limitations. Some methods produce faces that cannot be realistically animated because they do not model how wrinkles vary with expression. Other methods are trained on high-quality face scans and do not generalize well to in-the-wild images. We present the first approach that regresses 3D face shape and animatable details that are specific to an individual but change with expression. Our model, DECA (Detailed Expression Capture and Animation), is trained to robustly produce a UV displacement map from a low-dimensional latent representation that consists of person-specific detail parameters and generic expression parameters, while a regressor is trained to predict detail, shape, albedo, expression, pose and illumination parameters from a single image. To enable this, we introduce a novel detail-consistency loss that disentangles person-specific details from expression-dependent wrinkles. This disentanglement allows us to synthesize realistic person-specific wrinkles by controlling expression parameters while keeping person-specific details unchanged. DECA is learned from in-the-wild images with no paired 3D supervision and achieves state-of-the-art shape reconstruction accuracy on two benchmarks. Qualitative results on in-the-wild data demonstrate DECA's robustness and its ability to disentangle identity-and expression-dependent details enabling animation of reconstructed faces. The model and code are publicly available at https://deca.is.tue.mpg.de. CCS Concepts: • Computing methodologies → Mesh models.

Video-driven Neural Physically-based Facial Asset for Production

“…There are also works that successfully handle extreme skin deformation [Chen et al 2019;Wu et al 2016]. DECA [Feng et al 2021] further improve FLAME ] by additionally predicting individual-specific animatable details. High-fidelity facial details from real performers can be added to scanned facial models.…”

“…To the best of our knowledge, our approach is the first to provide video-driven facial geometries with dynamic physically-based texture assets. To demonstrate the video-driven performance of our approach, we compare against various video-driven facial reconstruction and animation methods, including parametric facial model DECA proposed by Feng et al [2021], a performer-specific method for production proposed by Laine et al [2017], and Deep Appearance Model proposed by Lombardi et al [2018]. We adopt the official pre-trained PyTorch model for the method of Feng et al [2021], and faithfully re-implement both the methods of Laine et al [2017] and Lombardi et al [2018] and train their models using the same training dataset as ours for a fair comparison.…”

“…To demonstrate the video-driven performance of our approach, we compare against various video-driven facial reconstruction and animation methods, including parametric facial model DECA proposed by Feng et al [2021], a performer-specific method for production proposed by Laine et al [2017], and Deep Appearance Model proposed by Lombardi et al [2018]. We adopt the official pre-trained PyTorch model for the method of Feng et al [2021], and faithfully re-implement both the methods of Laine et al [2017] and Lombardi et al [2018] and train their models using the same training dataset as ours for a fair comparison. Due to the lack of available code of the method of Moser et al [2021], we only compare the statistic results against several methods, including the one of Moser et al [2021], which demonstrates our approach achieves more comprehensive capabilities to reconstruct various facial attributes.…”

“…In particular, the data-driven strategy employed by learning-based techniques is applicable to handle datasets of different scales, as big as thousands of subjects for producing a generic model or as small as a single subject with different expressions and lighting conditions for portrait animation. For the former, generic models [Bao et al 2021;Feng et al 2021;Li et al 2020bRaj et al 2021] can achieve high robustness despite environment lighting changes and variations in facial geometry, motion or even physically-based attributes. The notable framework [Li et al 2020b] manages to learn personalized blend shapes along with pore-level dynamic physicallybased texture maps whereas latest advances [Bao et al 2021] can further generate riggable head models with high-resolution albedo and normal maps.…”

“…Qualitative comparison of video-driven result on in-the-wild monocular RGB inputs. (a) the input in-the-wild image, (b) predicted geometry of our method with normal map, before the refinement using in-the-wild images, (c) predicted geometry of our method with normal map, after the refinement using in-the-wild images, (d) predicted geometry ofLaine et al [2017], (e) predicted geometry ofLombardi et al [2018], before the refinement using in-the-wild images, (f) predicted geometry ofLombardi et al [2018], after the refinement using in-the-wild images, (g) predicted geometry ofFeng et al [2021], (h) predicted geometry ofGuo et al [2020]. With a quick refinement, our method can deal with head pose and lighting variance better than other methods.…”