Learning Individual Styles of Conversational Gesture

Abstract

Figure 1: Speech-to-gesture translation example. In this paper, we study the connection between conversational gesture and speech. Here, we show the result of our model that predicts gesture from audio. From the bottom upward: the input audio, arm and hand pose predicted by our model, and video frames synthesized from pose predictions using [10]. AbstractHuman speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-modal translation from "in-the-wild" monologue speech of a single speaker to their hand and arm motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and speech, we release a large video dataset of person-specific gestures.

Gesticulator: A framework for semantically-aware speech-driven gesture generation

“…Audio-Driven Gesture Generation. Most prior work on datadriven gesture generation has used the audio-signal as the only speech-input modality in the model [14,15,19,28,42]. For example, Sadoughi and Busso [42] trained a probabilistic graphical model to generate a discrete set of gestures based on the speech audiosignal, using discourse functions as constraints.…”

“…Kucherenko et al [28] extended this work by applying representation learning to the human pose and reducing the need for smoothing. Recently, Ginosar et al [15] applied a convolutional neural network with adversarial training to generate 2D poses from spectrogram features. However, driving either virtual avatars or humanoid robots requires 3D joint angles.…”

“…Like previous research in gesture generation [13,15], we represent speech audio by log-power mel-spectrogram features. For this, we extracted 64-dimensional acoustic feature vectors using a window length of 0.1 s and hop length 0.05 s (giving 20 fps).…”

“…We assessed the perceived human-likeness of the virtual character's motion and how the motion related to the character's speech using measures adapted from recent co-speech gesture generation papers [15,48]. Specifically, we asked the questions "In which video... ": (Q1) "...are the character's movements most human-like?"…”

Energy flows in gesture-speech physics: Exploratory findings and pre-registration of confirmatory analysis

“…Indeed, a recent breakthrough in deep-learning modeling suggests a highly invariant coupling of gesture and speech prosody. A recurrent neural network trained on person-specific gesture-speech sequences (motion and audio data from talk shows), was able to produce novel speech-synchronous gestures based on novel speech from the person the neural network was trained on (Ginosar et al, 2019). These neural networks are thus showing that there must be some person-specific invariant between speech acoustics and gesture motion, although it remains unknown what the neural network in fact picked up on in speech so as to produce gesture so well (but see Kucherenko, Hasegawa, Henter, Kaneko, & Kjellström, 2019).…”