A Comprehensive Review of Data‐Driven Co‐Speech Gesture Generation

Abstract

Gestures that accompany speech are an essential part of natural and efficient embodied human communication. The automatic generation of such co‐speech gestures is a long‐standing problem in computer animation and is considered an enabling technology for creating believable characters in film, games, and virtual social spaces, as well as for interaction with social robots. The problem is made challenging by the idiosyncratic and non‐periodic nature of human co‐speech gesture motion, and by the great diversity of communicative functions that gestures encompass. The field of gesture generation has seen surging interest in the last few years, owing to the emergence of more and larger datasets of human gesture motion, combined with strides in deep‐learning‐based generative models that benefit from the growing availability of data. This review article summarizes co‐speech gesture generation research, with a particular focus on deep generative models. First, we articulate the theory describing human gesticulation and how it complements speech. Next, we briefly discuss rule‐based and classical statistical gesture synthesis, before delving into deep learning approaches. We employ the choice of input modalities as an organizing principle, examining systems that generate gestures from audio, text and non‐linguistic input. Concurrent with the exposition of deep learning approaches, we chronicle the evolution of the related training data sets in terms of size, diversity, motion quality, and collection method (e.g., optical motion capture or pose estimation from video). Finally, we identify key research challenges in gesture generation, including data availability and quality; producing human‐like motion; grounding the gesture in the co‐occurring speech in interaction with other speakers, and in the environment; performing gesture evaluation; and integration of gesture synthesis into applications. We highlight recent approaches to tackling the various key challenges, as well as the limitations of these approaches, and point toward areas of future development.

Large language models in textual analysis for gesture selection

“…In this paper, we focus on the specification problem. For a review of the work on the animation problem see [54]. The specification problem can be approached with rule-based or data-driven approaches which the following sections discuss.…”

“…Because of this, data-driven gesture generation increasingly uses machine learning techniques, and specifically deep learning [1,54]. Many of these approaches use encode-decoder recurrent neural networks that learn the mapping from utterance text to gestures [6,72,73].…”

“…However, these approaches often still rely on gesture lexica [6] and in all cases, these data-driven approaches are dependent on the data they are trained on, with limited data sets available. Additionally, much of the semantic richness is often lost [54].…”

“…In essence, these approaches aim to generalize example gestures to semantically related utterances. However, the emphasis is increasingly shifting to use of state of the art deep learning approaches to generate gestures directly from the utterance, in part driven by the fact that larger language+gesture corpora have become available (see [54] for a comprehensive review).…”

Data-Driven Expressive 3D Facial Animation Synthesis for Digital Humans

“…As a result, new workflows for generating animation driven by data need to be proposed, implemented, and properly evaluated. Recent developments in deep learning techniques have proven to be particularly useful for such data-driven workflows for gesture synthesis [Nyatsanga et al 2023] and they hold immense potential for being applied to facial animation synthesis as well. More specifically, speech-driven facial animation synthesis is now being widely explored in both academic research [Cudeiro et al 2019;Fan et al 2021a;Xing et al 2023] and industry [JALI 2023].…”