Gesticulator: A framework for semantically-aware speech-driven gesture generation

Abstract

During speech, people spontaneously gesticulate, which plays a key role in conveying information. Similarly, realistic co-speech gestures are crucial to enable natural and smooth interactions with social agents. Current end-to-end co-speech gesture generation systems use a single modality for representing speech: either audio or text. These systems are therefore confined to producing either acoustically-linked beat gestures or semantically-linked gesticulation (e.g., raising a hand when saying "high"): they cannot appropriately learn to generate both gesture types. We present a model designed to produce arbitrary beat and semantic gestures together. Our deep-learning based model takes both acoustic and semantic representations of speech as input, and generates gestures as a sequence of joint angle rotations as output. The resulting gestures can be applied to both virtual agents and humanoid robots. Subjective and objective evaluations confirm the success of our approach. The code and video are available at the project page svito-zar.github.io/gesticulator.

A Large, Crowdsourced Evaluation of Gesture Generation Systems on Common Data: The GENEA Challenge 2020

“…Previous studies suggest that motion quality (human-likeness) may influence gesture appropriateness ratings in subjective evaluations [31,61]. Our experiments only partly managed to separate these two aspects of gesture perception.…”

“…The distance between speed histograms has also been used to evaluate gesture quality [29,31], since well-trained models should produce motion with similar properties to that of the actor it was trained on. In particular, it should have a similar motion-speed profile for any given joint.…”

No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures

“…Moving forward, neural networks were employed to predict a sequence of frames for gestures (Hasegawa et al, 2018), head motions (Sadoughi and Busso, 2018) and body motions (Shlizerman et al, 2018;Ahuja et al, 2019;Ginosar et al, 2019;Ferstl et al, 2019) conditioned on a speech input while Yoon et al (2019) uses only a text input. Unlike these approaches, Kucherenko et al (2020) rely on both speech and language for gesture generation. But their choice of early fusion to com-bine the modalities ignores multi-scale correlations (Tsai et al, 2019) between speech and language.…”

“…Expressivity Naturalness Relevance Timing S2G (Ginosar et al, 2019) 24.6 ± 3.1 22.1 ± 1.8 22.4 ± 1.7 27.6 ± 1.7 Gesticulator (Kucherenko et al, 2020) 31.9 ± 2.0 32.1 ± 1.7 31.4 ± 1.8 31.1 ± 1.7 Ours w/o G attn 35.0 ± 2.3 29.2 ± 1.7 30.9 ± 1.8 30.8 ± 1.7 Ours w/o AISLe 35.8 ± 2.9 35.7 ± 1.7 33.7 ± 1.7 32.1 ± 1.7 Ours 38.9 ± 1.7 36.7 ± 1.6 37.1 ± 1.7 35.3 ± 1.7…”

“…no multimodal attention block) and any form of re-sampling while training. Gesticulator (Kucherenko et al, 2020): Unlike MMS-Transformer , Gesticulator has a set of fully connected layer followed by autoregressive fully connected layers which are FiLM conditioned (Perez et al, 2018). In addition to audio and text, features of duration of each word (i.e.…”