No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures

Abstract

We study relationships between spoken language and co-speech gestures in context of two key challenges. First, distributions of text and gestures are inherently skewed making it important to model the long tail. Second, gesture predictions are made at a subword level, making it important to learn relationships between language and acoustic cues. We introduce Adversarial Importance Sampled Learning (or AISLe), which combines adversarial learning with importance sampling to strike a balance between precision and coverage. We propose the use of a multimodal multiscale attention block to perform subword alignment without the need of explicit alignment between language and acoustic cues. Finally, to empirically study the importance of language in this task, we extend the dataset proposed in Ahuja et al. (2020) with automatically extracted transcripts for audio signals. We substantiate the effectiveness of our approach through large-scale quantitative and user studies, which show that our proposed methodology significantly outperforms previous state-of-the-art approaches for gesture generation. Link to code, data and videos: https: //github.com/chahuja/aisle

Multimodal analysis of the predictability of hand-gesture properties

“…They may also scale better to large datasets. However, despite several attempts [1,26,53], there have in our view been no convincing demonstrations of recent data-driven approaches consistently generating gestures with a clear semantic relation to the speech content. For example, in terms of subjective gesture appropriateness for the speech, no system in the 2020 GENEA gesture-generation challenge [27] surpassed a bottom line that simply paired the input speech audio with mismatched excerpts of training data motion, completely unrelated to the speech.…”

“…While early hand gesture-generation systems mainly relied on rule-based approaches [6,24,37,40], data-driven gesture generation has become an important research area in recent years [1,13,26,53,54]. Both paradigms have advantages and disadvantages.…”

To Rate or Not To Rate: Investigating Evaluation Methods for Generated Co-Speech Gestures

“…When we interact with embodied conversational agents, we expect a similar manner of nonverbal communication as when interacting with humans. One way to achieve more human-like nonverbal behaviour in conversational agents is through the use of data-driven methods, which learn model parameters from data and gained in popularity over the past few years [1,17,18,40]. Data-driven methods have been used to generate lips synchronisation, eye gaze or facial expressions, however in this work we take co-speech gestures as a test bed for comparing evaluation methods.…”

“…Objective measures rely on an algorithmic approach to return a quantitative measure of the quality of the behaviour and are entirely automated, while subjective measures instead rely on ratings by human observers. Most recent papers on co-speech gesture generation report objective measures to assess the quality of the generated behaviour, with measures such as velocity diagrams or average jerk being popular [1,16,40]. These measures not only are easy to automate, but also allow comparisons across models.…”

“…We take generated co-speech gestures as a test bed for our evaluations but note that these findings may also apply to stimulus evaluation in other areas. Our hypotheses, design, and methodology were pre-registered before the data was gathered 1 . We present short video clips to human participants, with each video clip showing an avatar displaying combined verbal and nonverbal behaviour.…”