Predicting Head Pose from Speech with a Conditional Variational Autoencoder

Greenwood, David A.; Laycock, Stephen D.; Matthews, Iain

doi:10.21437/interspeech.2017-894

Cited by 39 publications

(29 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several recent works have used neural networks to generate bodymotion aspects such as locomotion [HHS * 17, HKS17, HAB19], lip movements [SSKS17] and head motion [GLM17,SB18]. A challenge in these domains is the large variation in the output given the same control.…”

Section: Data-driven Human Body-motion Generationmentioning

confidence: 99%

Style‐Controllable Speech‐Driven Gesture Synthesis Using Normalising Flows

Alexanderson

Henter

Kucherenko

et al. 2020

Computer Graphics Forum

134

150

View full text Add to dashboard Cite

Automatic synthesis of realistic gestures promises to transform the fields of animation, avatars and communicative agents. In off‐line applications, novel tools can alter the role of an animator to that of a director, who provides only high‐level input for the desired animation; a learned network then translates these instructions into an appropriate sequence of body poses. In interactive scenarios, systems for generating natural animations on the fly are key to achieving believable and relatable characters. In this paper we address some of the core issues towards these ends. By adapting a deep learning‐based motion synthesis method called MoGlow, we propose a new generative model for generating state‐of‐the‐art realistic speech‐driven gesticulation. Owing to the probabilistic nature of the approach, our model can produce a battery of different, yet plausible, gestures given the same input speech signal. Just like humans, this gives a rich natural variation of motion. We additionally demonstrate the ability to exert directorial control over the output style, such as gesture level, speed, symmetry and spacial extent. Such control can be leveraged to convey a desired character personality or mood. We achieve all this without any manual annotation of the data. User studies evaluating upper‐body gesticulation confirm that the generated motions are natural and well match the input speech. Our method scores above all prior systems and baselines on these measures, and comes close to the ratings of the original recorded motions. We furthermore find that we can accurately control gesticulation styles without unnecessarily compromising perceived naturalness. Finally, we also demonstrate an application of the same method to full‐body gesticulation, including the synthesis of stepping motion and stance.

show abstract

Section: Data-driven Human Body-motion Generationmentioning

confidence: 99%

Style‐Controllable Speech‐Driven Gesture Synthesis Using Normalising Flows

Alexanderson

Henter

Kucherenko

et al. 2020

Computer Graphics Forum

134

150

View full text Add to dashboard Cite

show abstract

“…Yoon et al [45] learned a mapping from text to gestures using a recurrent neural network. Speech-driven head-motion and facial gesture generation has been performed using methods such as Variational Autoencoders (VAEs) [31] to predict head pose conditioned on acoustic features [19], Bidirectional Long Short-Term Memory (BLSTM) networks [20,21,41], and conditional Generative Adversarial Networks (GANs) [18] as seen in [11,42]. In another line of work, Karras et al [28] trained a CNN-based neural network using speech together with a learned emotion representation as input to generate corresponding 3D meshes of faces with impressively little training data.…”

Section: Gesture Generationmentioning

confidence: 99%

Let's Face It

Jonell

Kucherenko

Henter

et al. 2020

Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

View full text Add to dashboard Cite

To enable more natural face-to-face interactions, conversational agents need to adapt their behavior to their interlocutors. One key aspect of this is generation of appropriate non-verbal behavior for the agent, for example facial gestures, here defined as facial expressions and head movements. Most existing gesture-generating systems do not utilize multi-modal cues from the interlocutor when synthesizing non-verbal behavior. Those that do, typically use deterministic methods that risk producing repetitive and non-vivid motions. In this paper, we introduce a probabilistic method to synthesize interlocutor-aware facial gestures ś represented by highly expressive FLAME parameters ś in dyadic conversations. Our contributions are: a) a method for feature extraction from multi-party video and speech recordings, resulting in a representation that allows for independent control and manipulation of expression and speech articulation in a 3D avatar; b) an extension to MoGlow, a recent motion-synthesis method based on normalizing flows, to also take multi-modal signals from the interlocutor as input and subsequently output interlocutor-aware facial gestures; and c) a subjective evaluation assessing the use and relative importance of the different modalities in the synthesized output. The results show that the model successfully leverages the input from the interlocutor to generate more appropriate behavior. Videos, data, and code are available at: https://jonepatr.github.io/lets_face_it/.

show abstract

“…More recently Haag [25] uses BLSTMs and Bottleneck features [26]. In our own earlier work [4], we use a BLSTM based Conditional Variational Autoencoder (CVAE) to model the many-to-many mapping of speech to head pose prediction, both for speaker, and for the head pose of the listener in dyadic conversation [27].…”

Section: Head Posementioning

confidence: 99%

“…Carrying on from our previous work [4], we use Deep BLSTMs to predict the facial deformation and the six DoF of rigid head pose combined. Clearly, much of the activity of the orofacial region has significant correspondence with speech production.…”

Section: Model Descriptionmentioning

confidence: 99%

“…In this work we model the complete facial activity during speech, along with the rigid pose of the speaker's head. We extend our earlier work on speaker head pose [4] by modelling six Degrees of Freedom (DoF): the rotations of nod, yaw and roll and the translations on those axes. Head pose has properties that make it difficult to model directly from speech.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Joint Learning of Facial Expression and Head Pose from Speech

Greenwood¹,

Matthews²,

Laycock³

2018

Interspeech 2018

Self Cite

View full text Add to dashboard Cite

Natural movement plays a significant role in realistic speech animation, and numerous studies have demonstrated the contribution visual cues make to the degree human observers find an animation acceptable. Natural, expressive, emotive, and prosodic speech exhibits motion patterns that are difficult to predict with considerable variation in visual modalities. Recently, there have been some impressive demonstrations of face animation derived in some way from the speech signal. Each of these methods have taken unique approaches, but none have included rigid head pose in their predicted output. We observe a high degree of correspondence with facial activity and rigid head pose during speech, and exploit this observation to jointly learn full face animation and head pose rotation and translation combined. From our own corpus, we train Deep Bi-Directional LSTMs (BLSTM) capable of learning long-term structure in language to model the relationship that speech has with the complex activity of the face. We define a model architecture to encourage learning of rigid head motion via the latent space of the speaker's facial activity. The result is a model that can predict lip sync and other facial motion along with rigid head motion directly from audible speech.

show abstract

Predicting Head Pose from Speech with a Conditional Variational Autoencoder

Cited by 39 publications

References 22 publications

Style‐Controllable Speech‐Driven Gesture Synthesis Using Normalising Flows

Style‐Controllable Speech‐Driven Gesture Synthesis Using Normalising Flows

Let's Face It

Joint Learning of Facial Expression and Head Pose from Speech

Contact Info

Product

Resources

About