Fast and deep facial deformations

Abstract

Film-quality characters typically display highly complex and expressive facial deformation. The underlying rigs used to animate the deformations of a character's face are often computationally expensive, requiring high-end hardware to deform the mesh at interactive rates. In this paper, we present a method using convolutional neural networks for approximating the mesh deformations of characters' faces. For the models we tested, our approximation runs up to 17 times faster than the original facial rig while still maintaining a high level of fidelity to the original rig. We also propose an extension to the approximation for handling high-frequency deformations such as fine skin wrinkles. While the implementation of the original animation rig depends on an extensive set of proprietary libraries making it difficult to install outside of an in-house development environment, our fast approximation relies on the widely available and easily deployed TensorFlow libraries. In addition to allowing high frame rate evaluation on modest hardware and in a wide range of computing environments, the large speed increase also enables interactive inverse kinematics on the animation rig. We demonstrate our approach and its applicability through interactive character posing and real-time facial performance capture.

Learning skeletal articulations with neural blend shapes

“…Bailey et al [2018] approximate the deformation of a complex production rig using neural networks, which reduces the execution cost and allows film-quality deformation in real-time applications. Later research extends this idea to more complicated facial rigs [Bailey et al 2020;Song et al 2020]. Neural networks can also be trained to convert high-level user control into rig parameters [Bailey et al 2020] to enable user-friendly editing of mesh deformation.…”

“…Later research extends this idea to more complicated facial rigs [Bailey et al 2020;Song et al 2020]. Neural networks can also be trained to convert high-level user control into rig parameters [Bailey et al 2020] to enable user-friendly editing of mesh deformation. train a graph neural network (GNN) to apply corrective displacements to linear deformations and create nonlinear effects.…”

A Revisit of Shape Editing Techniques: from the Geometric to the Neural Viewpoint

“…Bailey et al [244] propose a convolutional neural network for approximating facial deformations which can handle high-frequency deformations. The method pose, but also motion, so the regressor also uses the motion descriptor as input.…”

Semantic Deep Face Models

“…Another interesting contribution in neural semantic face modelling is the work of Bailey et. al [3], where semantic control over expression is achieved through rig parameters instead of blendweights. However, since their method is rig specific, and doesn't model appearance, it unfortunately cannot be used for several of the applications demonstrated in this work.…”

“…As we shall see in more detail in Section 2, recent methods have begun to investigate nonlinear face models using neural networks [28,1,14,20,16,24,3], which can, to some degree, overcome the limitations of linear models. Unfortunately, some of these approaches have thus far sacrificed the human interpretable nature of multi-linear models, as one typically loses semantics when moving to a latent space learned by a deep network.…”