Generating coherent spontaneous speech and gesture from text

Abstract

Embodied human communication encompasses both verbal (speech) and non-verbal information (e.g., gesture and head movements). Recent advances in machine learning have substantially improved the technologies for generating synthetic versions of both of these types of data: On the speech side, text-to-speech systems are now able to generate highly convincing, spontaneous-sounding speech using unscripted speech audio as the source material. On the motion side, probabilistic motion-generation methods can now synthesise vivid and lifelike speech-driven 3D gesticulation. In this paper, we put these two state-of-the-art technologies together in a coherent fashion for the first time. Concretely, we demonstrate a proof-of-concept system trained on a single-speaker audio and motion-capture dataset, that is able to generate both speech and full-body gestures together from text input. In contrast to previous approaches for joint speech-and-gesture generation, we generate full-body gestures from speech synthesis trained on recordings of spontaneous speech from the same person as the motion-capture data. We illustrate our results by visualising gesture spaces and textspeech-gesture alignments, and through a demonstration video. CCS CONCEPTS • Computing methodologies → Artificial intelligence.

Semantically Related Gestures Move Alike: Towards a Distributional Semantics of Gesture Kinematics

“…A humanoid can build its kinematic space from the bottom up, by detecting gestures in interaction, construct a kinematic similarity space over time, and infer from the distance matrices which gestures are likely to be semantically related (given the assumption that kinematic space and semantic space tend to align). Moreover, the humanoid's own gesture generation process may be tailored such that there is some weak dependency between the kinematics of gestures that are related in content, thus optimizing its gesture behavior to cohere in a similar way as human gesture does [36][37][38]. The current findings thus provide an exciting proof-of-concept that continuous communicative bodily movements that co-vary in kinematic structure, also co-vary in meaning.…”

Integrated Speech and Gesture Synthesis

“…However, the vast majority of these agents use an incoherent setup where the speech synthesis is trained on a different dataset than the gesture generation, and style and speaker identity may differ between components. In fact, the only system we are aware of where both components explicitly were trained on the same dataset is the one in [3], and we consequently use their approach as the baseline pipeline approach for our experiments in Section 5.…”

“…We only used the upper body data and removed the fingers (but not hand orientation) due to low capture accuracy there. For visualization, we instead used fixed, lightly-cupped hands on the avatar, similar to [3].…”

“…5.1.1 Baseline: Pipeline with Tacotron 2 and StyleGestures. We compare the models against a state-of-the-art pipeline system [3] which uses Tacotron 2 for speech synthesis and StyleGestures [2] for gesture generation. The only changes we made are using WaveGlow [42] for vocoding, as opposed to the Griffin-Lim algorithm used in [3], and only training on upper-body data.…”

“…We compare the models against a state-of-the-art pipeline system [3] which uses Tacotron 2 for speech synthesis and StyleGestures [2] for gesture generation. The only changes we made are using WaveGlow [42] for vocoding, as opposed to the Griffin-Lim algorithm used in [3], and only training on upper-body data. We used the publicly available, official implementation of StyleGestures 2 , using the recommended hyperparameters with 16 flow-steps and 512 channels in the LSTM layers.…”