Joint Learning of Facial Expression and Head Pose from Speech

Abstract

Natural movement plays a significant role in realistic speech animation, and numerous studies have demonstrated the contribution visual cues make to the degree human observers find an animation acceptable. Natural, expressive, emotive, and prosodic speech exhibits motion patterns that are difficult to predict with considerable variation in visual modalities. Recently, there have been some impressive demonstrations of face animation derived in some way from the speech signal. Each of these methods have taken unique approaches, but none have included rigid head pose in their predicted output. We observe a high degree of correspondence with facial activity and rigid head pose during speech, and exploit this observation to jointly learn full face animation and head pose rotation and translation combined. From our own corpus, we train Deep Bi-Directional LSTMs (BLSTM) capable of learning long-term structure in language to model the relationship that speech has with the complex activity of the face. We define a model architecture to encourage learning of rigid head motion via the latent space of the speaker's facial activity. The result is a model that can predict lip sync and other facial motion along with rigid head motion directly from audible speech.

MakeltTalk

“…Attention masks are used to focus on the most changing parts on the face, especially the lips. Greenwood et al [2018] jointly learnt facial expressions and head poses in terms of landmarks from a forked Bi-directional LSTM network. Most previous audio-to-face-animation work focused on matching speech content and left out style/identity information since the identity is usually bypassed due to mode collapse or averaging during training.…”

“…Last but not least, handling lip syncing and facial animation are not sufficient for the perception of realism of talking-heads. The entire facial expression considering the correlation between all facial elements and head pose also play an important role [Faigin 2012;Greenwood et al 2018]. These correlations, however, are less constrained by the audio and thus hard to be estimated.…”

MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement

“…These approaches achieve highly realistic results, but they are typically personalized and are not audio-driven. Most fully speech-driven 3D face animation techniques require either personalized models [5,22,26] or map to lower fidelity blendshape models [24] or facial landmarks [12,16]. Cao et al [5] propose speech-driven animation of a realistic textured personalized 3D face model that requires mocap data from the person to be animated, offline processing and blending of motion snippets.…”

Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

“…Head animation synthesis focuses on synthesizing head pose from input speech. Some works direct regress head pose with BiLSTM (Ding, Zhu, and Xie 2015;Greenwood, Matthews, and Laycock 2018) or the encoder of transformer (Vaswani et al 2017). More precisely, head pose generation from speech is a one-to-many mapping, Sadoughi and Busso (2018) employ GAN (Goodfellow et al 2014;Mirza and Osindero 2014;Yu et al 2019b,a) to retain the diversity.…”