Capture, Learning, and Synthesis of 3D Speaking Styles

Abstract

Input: speech signal and 3D templateOutput: 3D character animation Figure 1: Given an arbitrary speech signal and a static 3D face mesh as input (left), our model, VOCA outputs a realistic 3D character animation (right). Top: Winston Churchill. Bottom: Actor from Karras et al. [33]. See supplementary video. AbstractAudio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural network on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Character Animation) takes any speech signal as input-even speech in languages other than English-and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. VOCA also provides animator controls to alter speaking style, identity-dependent facial shape, and pose (i.e. head, jaw, and eyeball rotations) during animation. To our knowledge, VOCA is the only realistic 3D facial animation model that is read-ily applicable to unseen subjects without retargeting. This makes VOCA suitable for tasks like in-game video, virtual reality avatars, or any scenario in which the speaker, speech, or language is not known in advance. We make the dataset and model available for research purposes at

Using Facial Animation to Increase the Enfacement Illusion and Avatar Self-Identification

“…New commercial HMDs, such as the Vive Pro Eye, can enable correct rendering of the avatar motions, which may again improve the reported results. Additionally, lip-sync systems for avatar animation keep evolving and they are currently reaching human perception levels [6]. Hence we hypothesise that lip-sync will become an even more common form of facial animation.…”

3D Morphable Face Models—Past, Present, and Future

“…Some works aim to synthesize coherent dynamic 3D face videos of a fixed identity with the help of 3DMMs. These include works that synthesize 4D videos from a static 3D mesh paired with semantic label information [Bolkart and Wuhrer 2015a], and from a static 3D mesh and audio information [Cudeiro et al 2019].…”

Talking Face Generation with Expression-Tailored Generative Adversarial Network

“…VisemeNet [50], a threestage LSTM network is proposed to achieve real time audio-lip synchronization and can be seamlessly integrated into existing animation workflows. VOCA [13], which is trained on a unique 4D face dataset, takes any speech signal as input and realistically animates a wide range of adult faces. Methods based on computer graphics require collection and manipulation on complex head models.…”