Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance

Abstract

Gesture behavior is a natural part of human conversation. Much work has focused on removing the need for tedious hand-animation to create embodied conversational agents by designing speechdriven gesture generators. However, these generators often work in a black-box manner, assuming a general relationship between input speech and output motion. As their success remains limited, we investigate in more detail how speech may relate to different aspects of gesture motion. We determine a number of parameters characterizing gesture, such as speed and gesture size, and explore their relationship to the speech signal in a twofold manner. First, we train multiple recurrent networks to predict the gesture parameters from speech to understand how well gesture attributes can be modeled from speech alone. We find that gesture parameters can be partially predicted from speech, and some parameters, such as path length, being predicted more accurately than others, like velocity. Second, we design a perceptual study to assess the importance of each gesture parameter for producing motion that people perceive as appropriate for the speech. Results show that a degradation in any parameter was viewed negatively, but some changes, such as hand shape, are more impactful than others. A video summarization can be found at https://youtu.be/aw6-_5kmLjY. CCS CONCEPTS • Computing methodologies → Animation; Machine learning.

ExpressGesture: Expressive gesture generation from speech through database matching

“…15 One problem for machine learning approaches is the nondeterministic relationship of speech and gesture; a plethora of valid gestures can and do occur for a given utterance, hence modeling cospeech gestures as exact joint positions or angles may be too constrictive to capture natural variety. Recent work instead proposes modeling the speech-gesture relation with higher-level parameters; Ferstl et al 1 propose a set expressive gesture parameters, namely, velocity, initial acceleration, gesture size, arm swivel, and hand opening, and show they can both be estimated from speech as well as perceptually impacting speech-gesture match. In a preliminary study, replacing GT gestures with parameter-matched gestures of similar length was shown to outperform unmatched gestures.…”

“…Accompanying this work, we will release our full database of gestures, together with their values for the five expressive parameters, and corresponding speech. 1 We also release the five trained speech-to-parameter models for reproducibility (step 4 in Figure 1). We think this database will be a valuable resource for gesture research.…”

“…Theoretically, we can use any parameter automatically computable from a motion segment, but choose the five gesture parameters that were shown previously to be associated with the speech signal as well as impacting perceptions of the quality of the speech-gesture match: 1 (1) Gesture velocity, (2) the mean acceleration to the first major velocity peak, (3) gesture size measured by the total path length, (4) arm swivel (bringing the elbow closer or further from the body), and (5) hand opening (calculated as the mean distance of the finger tips from the base of the hand). The five parameters are estimated from the GeMAPS prosodic speech features, 25 computed automatically using openSMILE, 26 a process described in detail in Ferstl et al 1 Given the gesture parameter values, the database is searched for the best match, as determined by a match rank. A gesture's match rank for one parameter is given by the relative similarity to the desired parameter value.…”

“…Each gesture receives five rank values, one for each parameter. Rank values are weighted based on the findings of Ferstl et al 1 of how well a parameter can be predicted from speech and its perceptual importance for speech-gesture match. For example, gesture swivel and size are predicted well from speech, and acceleration is predicted better than velocity; hand shape has a strong perceptual impact on the perceived gesture match.…”

“…Instead of modeling an implicit relationship of exact, high-dimensional joint poses and the speech signal, we use a higher level representation of the gesture motion through expressive parameters shown in previous work to be associated with the speech signal as well as being perceptually important to the quality of the speech-gesture match. 1 These gesture parameters are gesture velocity, acceleration, size, arm swivel angle, and extent of hand opening.…”