Gesture modeling and animation based on a probabilistic re-creation of speaker style

Abstract

Animated characters that move and gesticulate appropriately with spoken text are useful in a wide range of applications. Unfortunately, this class of movement is very difficult to generate, even more so when a unique, individual movement style is required. We present a system that, with a focus on arm gestures, is capable of producing full-body gesture animation for given input text in the style of a particular performer. Our process starts with video of a person whose gesturing style we wish to animate. A tool-assisted annotation process is performed on the video, from which a statistical model of the person's particular gesturing style is built. Using this model and input text tagged with theme, rheme and focus, our generation algorithm creates a gesture script. As opposed to isolated singleton gestures, our gesture script specifies a stream of continuous gestures coordinated with speech. This script is passed to an animation system, which enhances the gesture description with additional detail. It then generates either kinematic or physically simulated motion based on this description. The system is capable of generating gesture animations for novel text that are consistent with a given performer's style, as was successfully validated in an empirical user study.

Don’t Scratch! Self-adaptors Reflect Emotional Stability

“…Software based on [25,28] was used to generate the accompanying animation clips. Both the communicative gestures and self-adaptors were generated by editing sampled motion data and these can be controlled independently.…”

A Perceptual Study of the Relationship between Posture and Gesture for Virtual Characters

“…Kopp and Wachsmuth [3] designed a system that generates gesture, facial behaviors and speech from XMLbased descriptions and synchronizes them with co-articulation and transition effects. Neff et al [4] generated gestures for a conversational agent based on learned statistical models from video analysis. Levine et al [14,15] animated gestures by learning hidden structure between acoustic features and body language from motion capture data.…”

“…In particular, we do not understand which aspects of body motion are perceptually relevant for gestures. In interactive character applications, gestures must often be generated or adapted to match the character's selected speech [1][2][3][4]. How can body motion be adjusted to match these gestures?…”

Systematicity and Idiosyncrasy in Iconic Gesture Use: Empirical Analysis and Computational Modeling

“…Third, an adequate model of how speakers produce iconic gestures must account for both types of influential patterns, general and individual ones. Previous modeling attempts either ignored idiosyncrasy coming up with generalized model-based approaches [18], or they employ statistical data-driven techniques which have problems with identifying and explicating systematicities from corpora of managable size [27]. We have proposed and described elsewhere [17,4] a production architecture that is inspired by psycholinguistic models [16,7] and accounts for our first two requirements.…”

“…Based on these findings, we describe in Section 4 a computational modeling account that goes beyond previous systems, which either rely on generalized rule-based models that disregard idiosyncrasy in gesture use [6,18], or employ data-based methods that approximate single speakers but have difficulties with extracting systematicities of gesture use. These data-based approaches are typically (and successfully) employed to generate gesturing behavior which has no particular meaning-carrying function, e.g., discourse gestures [27] or beat gestures (Theune & Brandhorst, this volume). We propose to combine probabilistic and rule-based decision-making.…”