High-Quality Face Capture Using Anatomical Muscles

Bao, Michael; Cong, Matthew; Grabli, Stéphane; Fedkiw, Ronald

doi:10.1109/cvpr.2019.01106

Cited by 16 publications

(7 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This has been traditionally addressed by recovering a personalized set of expression bases known as blendshapes, obtained through deformation transfer [7,19,21,22,46] or deep neural networks [1,9,58]. Other rigging techniques have also been investigated, such as joint-based representations [17,21,54] and muscle-based systems [2]. To represent detailed geometry, a few methods additionally reconstruct a layer of mid-or fine-level correctives that can be deformed along with the underlying coarse mesh [17,19,22,58].…”

Section: Related Workmentioning

confidence: 99%

I M Avatar: Implicit Morphable Head Avatars from Videos

Zheng¹,

Abrevaya²,

Bühler³

et al. 2021

Preprint

View full text Add to dashboard Cite

Traditional morphable face models provide fine-grained control over expression but cannot easily capture geometric and appearance details. Neural volumetric representations approach photo-realism but are hard to animate and do not generalize well to unseen expressions. To tackle this problem, we propose IMavatar (Implicit Morphable avatar), a novel method for learning implicit head avatars from monocular videos. Inspired by the fine-grained control mechanisms afforded by conventional 3DMMs, we represent the expression-and pose-related deformations via learned blendshapes and skinning fields. These attributes are pose-independent and can be used to morph the canonical geometry and texture fields given novel expression and pose parameters. We employ ray tracing and iterative rootfinding to locate the canonical surface intersection for each pixel. A key contribution is our novel analytical gradient formulation that enables end-to-end training of IMavatars from videos. We show quantitatively and qualitatively that our method improves geometry and covers a more complete expression space compared to state-of-the-art methods.

show abstract

Section: Related Workmentioning

confidence: 99%

I M Avatar: Implicit Morphable Head Avatars from Videos

Zheng¹,

Abrevaya²,

Bühler³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…[40] addressed a similar problem where they solved for a set of parameters that determined blendshape muscle geometry such that attached zero-length springs drove their quasistatic simulation mesh to well match a ground truth target. They utilized the approach in [38] to evaluate search directions for the optimization (noting that [38] solved for muscle activations directly whereas [40] obtained activations indirectly using zero-length springs attached to kinematically driven geometry as proposed in [39]). Although the material model forces only depend on r * unlike the zero length springs that depend on both r * andr, r * depends onr and so the dependencies may be written as f Z (r * (r),r) and f M (r * (r)).…”

Section: Buckling and Inextensibility Priormentioning

confidence: 99%

“…Conceptually speaking, the goal is to find a set of network parameters w that determine cloth geometry via the network in [32] such that the zero-length springs attached to that network driven cloth geometry drive the quasistatic simulation mesh to well match the training data. [40] addressed a similar problem where they solved for a set of parameters that determined blendshape muscle geometry such that attached zero-length springs drove their quasistatic simulation mesh to well match a ground truth target. They utilized the approach in [38] to evaluate search directions for the optimization (noting that [38] solved for muscle activations directly whereas [40] obtained activations indirectly using zero-length springs attached to kinematically driven geometry as proposed in [39]).…”

Section: Buckling and Inextensibility Priormentioning

confidence: 99%

“…This is quite important because even with a fast solver, the second order cone program approach in Section 3 and 4 is 40 to 50 times slower than the quasistatic approach in Section 5 and 6. Regarding the quasistatic case, the derivatives are more straight-forward, although care must be taken for poor conditioning and inversion; however, this has all been previously addressed some time ago, see [36,37,38] which require only minor modifications (as discussed in [39,40]) for our purposes.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Coercing machine learning to output physically accurate results

Geng

Johnson

Fedkiw

2020

Journal of Computational Physics

Self Cite

View full text Add to dashboard Cite

Many machine/deep learning artificial neural networks are trained to simply be interpolation functions that map input variables to output values interpolated from the training data in a linear/nonlinear fashion. Even when the input/output pairs of the training data are physically accurate (e.g. the results of an experiment or numerical simulation), interpolated quantities can deviate quite far from being physically accurate. Although one could project the output of a network into a physically feasible region, such a postprocess is not captured by the energy function minimized when training the network; thus, the final projected result could incorrectly deviate quite far from the training data. We propose folding any such projection or postprocess directly into the network so that the final result is correctly compared to the training data by the energy function. Although we propose a general approach, we illustrate its efficacy on a specific convolutional neural network that takes in human pose parameters (joint rotations) and outputs a prediction of vertex positions representing a triangulated cloth mesh. While the original network outputs vertex positions with erroneously high stretching and compression energies, the new network trained with our physics prior remedies these issues producing highly improved results. there will be large errors inf (w, x T ) when compared to y T . On the other hand, although one could create a network with many degrees of freedom in order to capture y T =f (w, x T ) as accurately as desired, even exactly,f (w, x) could oscillate wildly and inaccurately when x is not equal to x T , i.e. overfitting. See, e.g. [14,15,16,17]. One needs to take great care when designing the network architecture in order to avoid underfitting while still allowing for enough regularization to also avoid overfitting. Likewise, the form of the energy function and nature of the numerical optimization techniques also need careful consideration. Some of the most popular methods include variants of BFGS [18,19,20] and a number of methods based on gradient descent [21,22] (see also [23] and the references therein) or interpreting gradient descent as a numerical approximation to an ordinary differential equation to be solved via various approaches motivated by order of accuracy [24,25] and adaptive time-stepping [26,27,28,29,30].Devising a network architecture with enough representative capability to alleviate underfitting while still being amenable to the regularization required to avoid overfitting, and subsequently applying numerical optimization techniques to an adequately designed energy in order to find reasonable parameters w is a quite difficult and mostly experimental endeavour. Thus, much of the progress made by the community emanates from the laborious creation of data sets that many researchers can consider in order to design network architectures and find suitable parameters w, see e.g. [31]. This is typically driven by a community (rather than an individual or group) effort, and state-of-the-art r...

show abstract

“…The visual analysis of human motion has broad application prospects in human-computer interaction, video conferencing, medical diagnosis, virtual reality, etc., which makes it a frontier direction that has attracted the attention of researchers in recent years [7]. The main purpose of visual analysis is to detect, identify, and track the human body from a set of image sequences containing people and to understand and describe its behavior.…”

Section: Introductionmentioning

confidence: 99%

Extraction of Human Motion Information from Digital Video Based on 3D Poisson Equation

Wang

Chang

2021

Advances in Mathematical Physics

View full text Add to dashboard Cite

Based on the 3D Poisson equation, this paper extracts the features of the digital video human body action sequence. By solving the Poisson equation on the silhouette sequence, the time and space features, time and space structure features, shape features, and orientation features can be obtained. First, we use the silhouette structure features in three-dimensional space-time and the orientation features of the silhouette in three-dimensional space-time to represent the local features of the silhouette sequence and use the 3D Zernike moment feature to represent the overall features of the silhouette sequence. Secondly, we combine the Bayesian classifier and AdaBoost classifier to learn and classify the features of human action sequences, conduct experiments on the Weizmann video database, and conduct multiple experiments using the method of classifying samples and selecting partial combinations for training. Then, using the recognition algorithm of motion capture, after the above process, the three-dimensional model is obtained and matched with the model in the three-dimensional model database, the sequence with the smallest distance is calculated, and the corresponding skeleton is outputted as the results of action capture. During the experiment, the human motion tracking method based on the university matching kernel (EMK) image kernel descriptor was used; that is, the scale invariant operator was used to count the characteristics of multiple training images, and finally, the high-dimensional feature space was mapped into the low-dimensional to obtain the feature space approximating the Gaussian kernel. Based on the above analysis, the main user has prior knowledge of the network environment. The experimental results show that the method in this paper can effectively extract the characteristics of human body movements and has a good classification effect for bending, one-foot jumping, vertical jumping, waving, and other movements. Due to the linear separability of the data in the kernel space, fast linear interpolation regression is performed on the features in the feature space, which significantly improves the robustness and accuracy of the estimation of the human motion pose in the image sequence.

show abstract

High-Quality Face Capture Using Anatomical Muscles

Cited by 16 publications

References 52 publications

I M Avatar: Implicit Morphable Head Avatars from Videos

I M Avatar: Implicit Morphable Head Avatars from Videos

Coercing machine learning to output physically accurate results

Extraction of Human Motion Information from Digital Video Based on 3D Poisson Equation

Contact Info

Product

Resources

About