Dynamic facial asset and rig generation from a single scan

Abstract

The creation of high-fidelity computer-generated (CG) characters for films and games is tied with intensive manual labor, which involves the creation of comprehensive facial assets that are often captured using complex hardware. To simplify and accelerate this digitization process, we propose a framework for the automatic generation of high-quality dynamic facial models, including rigs which can be readily deployed for artists to polish. Our framework takes a single scan as input to generate a set of personalized blendshapes, dynamic textures, as well as secondary facial components ( e.g. , teeth and eyeballs). Based on a facial database with over 4, 000 scans with pore-level details, varying expressions and identities, we adopt a self-supervised neural network to learn personalized blendshapes from a set of template expressions. We also model the joint distribution between identities and expressions, enabling the inference of a full set of personalized blendshapes with dynamic appearances from a single neutral input scan. Our generated personalized face rig assets are seamlessly compatible with professional production pipelines for facial animation and rendering. We demonstrate a highly robust and effective framework on a wide range of subjects, and showcase high-fidelity facial animations with automatically generated personalized dynamic textures.

Video-driven Neural Physically-based Facial Asset for Production

“…Lattas et al [2021] manage to infer renderable photorealistic 3D faces from a single image. Since facial animation corresponds to a series of 3D scans, we can predict blendshapes from a single 3D scan of the neutral expression of the performer and then infer dynamic texture maps in a generative manner based on expression offsets [Li et al 2020b]. Differently, we build dynamic textures using wrinkle maps, which allows us to maintain the authenticity of the high-resolution textures while avoiding the artifacts generated by super-resolution networks.…”

“…Recent deep learning techniques [Lattas et al 2021;Li et al 2020b,a] have shown hopeful results to enable automated facial asset generation direct from the captured data with little human intervention, bringing huge potential to revolutionize production-level workflows. In particular, the data-driven strategy employed by learning-based techniques is applicable to handle datasets of different scales, as big as thousands of subjects for producing a generic model or as small as a single subject with different expressions and lighting conditions for portrait animation.…”

“…In particular, the data-driven strategy employed by learning-based techniques is applicable to handle datasets of different scales, as big as thousands of subjects for producing a generic model or as small as a single subject with different expressions and lighting conditions for portrait animation. For the former, generic models [Bao et al 2021;Feng et al 2021;Li et al 2020bRaj et al 2021] can achieve high robustness despite environment lighting changes and variations in facial geometry, motion or even physically-based attributes. The notable framework [Li et al 2020b] manages to learn personalized blend shapes along with pore-level dynamic physicallybased texture maps whereas latest advances [Bao et al 2021] can further generate riggable head models with high-resolution albedo and normal maps.…”

“…For the former, generic models [Bao et al 2021;Feng et al 2021;Li et al 2020bRaj et al 2021] can achieve high robustness despite environment lighting changes and variations in facial geometry, motion or even physically-based attributes. The notable framework [Li et al 2020b] manages to learn personalized blend shapes along with pore-level dynamic physicallybased texture maps whereas latest advances [Bao et al 2021] can further generate riggable head models with high-resolution albedo and normal maps. Their generated assets are directly usable to existing production pipelines for producing facial animations and photo-realistic rendering.…”

EmoGen: Quantifiable Emotion Generation and Analysis for Experimental Psychology

“…Hao Li and coleagues [24] pioneer example-based rigging from a handful of training poses of the new identity. The most impressive advance in the field is the work of Jiaman Li and colleagues [25] who generate personalised blendshapes with the corresponding dynamic textures from a single neutral scan. Their approach is based on training two cascaded neural networks, specifically one for the personalised blendshape generation followed by another for the generation of dynamic texture maps (albedo, specular intensity, displacement), tailored to the expression geometries.…”