ZeroEGGS: Zero‐shot Example‐based Gesture Generation from Speech

Abstract

We present ZeroEGGS, a neural network framework for speech-driven gesture generation with zero-shot style control by example. This means style can be controlled via only a short example motion clip, even for motion styles unseen during training. Our model uses a Variational framework to learn a style embedding, making it easy to modify style through latent space manipulation or blending and scaling of style embeddings. The probabilistic nature of our framework further enables the generation of a variety of outputs given the input, addressing the stochastic nature of gesture motion. In a series of experiments, we first demonstrate the flexibility and generalizability of our model to new speakers and styles. In a user study, we then show that our model outperforms previous state-of-the-art techniques in naturalness of motion, appropriateness for speech, and style portrayal. Finally, we release a high-quality dataset of full-body gesture motion including fingers, with speech, spanning across 19 different styles. Our code and data are publicly available at https:// github.com/ ubisoft/ ubisoft-laforge-ZeroEGGS.

A Comprehensive Review of Data‐Driven Co‐Speech Gesture Generation

“…In a similar vein, Ghorbani et al [GFC22, GFH*23], used a VAE‐based framework for style controllable co‐speech gesture generation conditioned by a zero‐shot motion example i.e., an instance of a motion style unseen during training. Given an audio input and a motion example, they generated an encoding of the audio and a style embedding from the motion, and the two latent codes were used to guide the generation of stylized gestures.…”

“…Style specification is also not data efficient, requiring as many samples as the size of the training set for the model to learn a style [AHKB20, ALNM20]. We conclude this section by discussing several works that proposed approaches for data‐efficient style specification [GFC22, GFH*23, FGPO22, ALM22].…”

“…A deeper analysis of the impact of speaker identity input [KNN*22] shows that different speakers have different gesture‐property prediction certainty, evoking even more interest in the idiosyncrasies of co‐speech gestures. More recently, it was also shown that a short motion clip can be used for style control in “zero‐shot style adaptation” [FGPO22, GFC22, GFH*23]. For many applications, it is desirable for the designer to be able to control the nature of the motion.…”

“…Ghorbani et al [GFC22, GFH*23] proposed a framework that improves on high‐level style portrayal by using exemplar motion sequences that demonstrate the intended stylistic expression of gesture motion. Their framework was able to efficiently extract style parameters in a zero‐shot manner, only requiring a single example motion and was able to generalize to example motions (and therefore styles) unseen during training.…”

Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer

“…Compared to CNN and RNN based models, the transformer model [Vaswani et al 2017] is relatively less explored in the audio-driven motion synthesis. Saeed et al [2023] present a variational transformer for encoding style information, whereas they adopt recurrent networks to model motion generation from both speech and style. Valle-Pérez et al [2021] and Li et al [2021b] propose generative transformer approaches with normalizing flow for dancing motion synthesis from music.…”