Learning to Generate Customized Dynamic 3D Facial Expressions

Potamias, Rolandos Alexandros; Zheng, Jiantao; Ploumpis, Stylianos; Bouritsas, Giorgos; Ververas, Evangelos; Zafeiriou, Stefanos

doi:10.1007/978-3-030-58526-6_17

Cited by 12 publications

(4 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In applications where clear non-rigid variations are concerned (e.g. facial expressions and body movements), however, illustrating the superiority of neural models is more straightforward [19] [41]. Moreover, training a single fullface GE, trained on an NVIDIA GeForce RTX 2080 Ti using PyTorch 1.1.0 takes 18 minutes and 25 seconds, while the PCA method takes 13 seconds in MATLAB 2020, on the same machine.…”

Section: Discussionmentioning

confidence: 99%

Multi-Scale Part-Based Syndrome Classification of 3D Facial Images

et al. 2022

Self Cite

View full text Add to dashboard Cite

Identification and delineation of craniofacial characteristics support the clinical and molecular diagnosis of genetic syndromes. Deep learning (DL) frameworks for syndrome identification from 2D facial images are trained on large clinical datasets using standard convolutional neural networks for classification. In contrast, despite the increased availability of 3D scanners in clinical setups, similar frameworks remain absent for 3D facial photographs. The main challenges involve working with smaller datasets and the need for DL operations applicable to 3D geometric data. Therefore, to date, most 3D methods refrain from working across multiple syndromic groups and/or are solely based on traditional machine learning. The first contribution of this work is the use of geometric deep learning with spiral convolutions in a triplet-loss architecture. This geometric encoding (GE) learns a lower dimensional metric space from 3D facial data that is used as input to linear discriminant analysis (LDA) performing multiclass classification. Benchmarking is done against principal component analysis (PCA), a common technique in 3D facial shape analysis, and related work based on 65 distinct 3D facial landmarks as input to LDA. The second contribution of this work involves a part-based implementation to 3D facial shape analysis and multi-class syndrome classification, and this is applied to both GE and PCA. Based on 1,786 3D facial photographs of controls and individuals from 13 different syndrome classes, a five-fold cross-validation was used to investigate both contributions. Results indicate that GE performs better than PCA as input to LDA, and this especially so for more compact (lower dimensional) spaces. In addition, a part-based approach increases performance significantly for both GE and PCA, with a more significant improvement for the latter. I.e., this contribution enhances the power of the dataset. Finally, and interestingly, according to ablation studies within the part-based approach, the upper lip is the most distinguishing facial segment for classifying genetic syndromes in our dataset, which follows clinical expectation. This work stimulates an enhanced use of advanced part-based geometric deep learning methods for 3D facial imaging in clinical genetics.

show abstract

Section: Discussionmentioning

confidence: 99%

Multi-Scale Part-Based Syndrome Classification of 3D Facial Images

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…As future work we plan to improve the minimal motion trembling in the generated sequences of the proposed selfsupervised audio-informed architecture, by employing an additional temporal coherence loss term that computes the distances between adjacent frames on ground truth and generated sequences, similar to [69], [70]. Currently, we are experimenting with different pose normalization techniques and sequences with lower frame rates, in addition to new approaches for conditioning the dance synthesis process on various low-level musical and audio features.…”

Section: Discussionmentioning

confidence: 99%

DanceConv: Dance Motion Generation With Convolutional Networks

et al. 2022

View full text Add to dashboard Cite

Automatically synthesizing dance motion sequences is an increasingly popular research task in the broader field of human motion analysis. Recent approaches have mostly used recurrent neural networks (RNNs), which are known to suffer from prediction error accumulation, usually limiting models to synthesize short choreographies of less than 100 poses. In this paper we present a multimodal convolutional autoencoder that combines 2D skeletal and audio information by employing an attention-based feature fusion mechanism, capable of generating novel dance motion sequences of arbitrary length. We first validate the ability of our system to capture the temporal context of dancing in a unimodal setting, by considering only skeletal features as input. According to 1440 rating answers provided by 24 participants in our initial user-study, we show that the optimal performance was presented by the model that was trained with input sequences of 500 poses. Based on this outcome, we train the proposed multimodal architecture with two different approaches, namely teacher-forcing and self-supervised curriculum learning, to deal with the autoregressive error accumulation phenomenon. In our evaluation campaign, we generate 1800 sequences and compare our method against two state-of-the-art approaches. Through qualitative and quantitative experiments we demonstrate the improvements introduced by the proposed multimodal architecture in terms of realism, motion diversity and multimodality, reducing the Fréchet Inception Distance (FID) metric value by 0.39. Subjective results confirm the effectiveness of our approach to synthesize diverse dance motion sequences, reporting a 6% increase in style consistency preference according to 1800 answers provided by 45 evaluators.

show abstract

“…In order to use Neural3DMM as a fitting method, we used the modified architecture as defined in [20], where the model was trained to generate an expressive mesh given its neutral counterpart and the target landmarks Z e as input. The mean per-vertex Euclidean error between the reconstructed meshes and their ground truth was used as measure, as in the majority of works [3], [10], [24], [25].…”

Section: A 3d Expression Fittingmentioning

confidence: 99%

The Florence 4D Facial Expression Dataset

Principi¹,

Ferrari²,

Otberdout³

et al. 2022

Preprint

View full text Add to dashboard Cite

Human facial expressions change dynamically, so their recognition / analysis should be conducted by accounting for the temporal evolution of face deformations either in 2D or 3D. While abundant 2D video data do exist, this is not the case in 3D, where few 3D dynamic (4D) datasets were released for public use. The negative consequence of this scarcity of data is amplified by current deep learning based-methods for facial expression analysis that require large quantities of variegate samples to be effectively trained. With the aim of smoothing such limitations, in this paper we propose a large dataset, named Florence 4D, composed of dynamic sequences of 3D face models, where a combination of synthetic and real identities exhibit an unprecedented variety of 4D facial expressions, with variations that include the classical neutralapex transition, but generalize to expression-to-expression. All these characteristics are not exposed by any of the existing 4D datasets and they cannot even be obtained by combining more than one dataset. We strongly believe that making such a data corpora publicly available to the community will allow designing and experimenting new applications that were not possible to investigate till now. To show at some extent the difficulty of our data in terms of different identities and varying expressions, we also report a baseline experimentation on the proposed dataset that can be used as baseline.

show abstract

Learning to Generate Customized Dynamic 3D Facial Expressions

Cited by 12 publications

References 39 publications

Multi-Scale Part-Based Syndrome Classification of 3D Facial Images

Multi-Scale Part-Based Syndrome Classification of 3D Facial Images

DanceConv: Dance Motion Generation With Convolutional Networks

The Florence 4D Facial Expression Dataset

Contact Info

Product

Resources

About