SPECTRE: Visual Speech-Informed Perceptual 3D Facial Expression Reconstruction from Videos

Filntisis, Panagiotis P.; Retsinas, George; Paraperas-Papantoniou, Foivos; Katsamanis, Athanasios; Roussos, Anastasios; Maragos, Petros

doi:10.1109/cvprw59228.2023.00609

Cited by 11 publications

(2 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent studies utilize deep learning frameworks based on self-supervised learning to predict 3DMM parameters from input images. They can create plausible 3D face without ground-truth 3D facial scan data by employing various loss functions, such as the landmark reprojection loss, photometric loss, and face recognition loss, to train the deep neural networks [1,[10][11][12][13].…”

Section: Introductionmentioning

confidence: 99%

“…Recently, various new loss functions and architectures have been introduced to address the limitations of existing methods with respect to reconstruction accuracy of the rich and detailed facial expressions [12,13,46,47]. In particular, the method of capturing emotions and reconstructing them into 3D faces demonstrates notable efficacy [12].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Action Unit-Based 3D Face Reconstruction Using Transformers

Kim,

Wang,

Lee

2024

Preprint

View full text Add to dashboard Cite

The reconstruction of 3D face shapes and expressions from single 2D images remains unconquered due to the lack of detailed modeling of human facial movements such as the correlation between the different parts of faces. Facial action units (AUs), which represent detailed taxonomy of the human facial movements based on observation of activation of muscles or muscle groups, can be used to model various facial expression types. We present a novel 3D face reconstruction framework called AU feature-based 3D FAce Reconstruction using Transformer (AUFART) that can generate a 3D face model that is responsive to AU activation given a single monocular 2D image to capture expressions. AUFART leverages AU-specific features as well as facial global features to achieve accurate 3D reconstruction of facial expressions using transformers. We also introduce a loss function which is to force the learning toward the minimal discrepancy in AU activations between the input and rendered reconstruction. The proposed framework achieves an average F1 score of 0.39, outperforming state-of-the-art methods.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Action Unit-Based 3D Face Reconstruction Using Transformers

Kim,

Wang,

Lee

2024

Preprint

View full text Add to dashboard Cite

show abstract

3D facial animation driven by speech-video dual-modal signals

Ji,

Liao,

Dong

et al. 2024

Complex Intell. Syst.

View full text Add to dashboard Cite

In recent years, the applications of digital humans have become increasingly widespread. One of the most challenging core technologies is the generation of highly realistic and automated 3D facial animation that combines facial movements and speech. The single-modal 3D facial animation driven by speech typically ignores the weak correlation between speech and upper facial movements as well as head posture. In contrast, the video-driven approach can perfectly solve the posture problem while obtaining natural expressions. However, mapping 2D facial information to 3D facial information may lead to information loss, which make lip synchronization generated by video-driven methods is not as good as the speech-driven methods trained on 4D facial data. Therefore, this paper proposes a dual-modal generation method that uses speech and video information to generate more natural and vivid 3D facial animation. Specifically, the lip movements related to speech are generated by speech-video information, while speech-uncorrelated postures and expressions are generated solely by video information. The speech-driven module is used to extract speech features, and its output lip animation is then used as the foundation for facial animation. The expression and pose module is used to extract temporal visual features for regressing expression and head posture parameters. We fuse speech and video features to obtain chin posture parameters related to lip movements, and use these parameters to fine-tune the lip animation generated form the speech-driven module. This paper introduces multiple consistency losses to enhance the network’s capability to generate expressions and postures. Experiments conducted on the LRS3, TCD-TIMIT and MEAD datasets show that the proposed method achieves better performance on evaluation metrics such as CER, WER, VER and VWER than the current state-of-the-art methods. In addition, a perceptual user study show that over 77% and 70% of cases believe that this paper’s method is better than the comparative algorithms EMOCA and SPECTRE in terms of realism. In terms of lip synchronization, it received over 79% and 66% of cases support, respectively. Both evaluation methods demonstrate the effectiveness of the proposed method.

show abstract