Visual prosody: facial movements accompanying speech

Abstract

As we articulate speech, we usually move the head and exhibit various facial expressions. This visual aspect of speech aids understanding and helps communicating additional information, such as the speaker's mood. In this paper we analyze quantitatively head and facial movements that accompany speech and investigate how they relate to the text's prosodic structure. We recorded several hours of speech and measured the locations of the speakers' main facial features as well as their head poses. The text was evaluated with a prosody prediction tool, identifying phrase boundaries and pitch accents. Characteristic for most speakers are simple motion patterns that are repeatedly applied in synchrony with the main prosodic events. Direction and strength of head movements vary widely from one speaker to another, yet their timing is typically well synchronized with the spoken text. Understanding quantitatively the correlations between head movements and spoken text is important for synthesizing photo-realistic talking heads. Talking heads appear much more engaging when they exhibit realistic motion patterns.

Head and facial gestures synthesis using PAD model for an expressive talking avatar

“…The prosodic word carries both prosodic structure and semantic meaning in Chinese spoken language. Early study has found that head movement are strongly correlated with the prosodic structure of the text [18]. In our data, we have observed the head moves periodically and is nearly synchronous with the segmentation of prosodic word.…”

“…The first assumption has been validated in Section 3.1. The second assumption of head shake motion pattern during speech has also been observed in previous study [18,20]. A sinusoidal function is proposed to generate head motion sequence within prosodic word based on the above assumptions.…”

“…Previous studies measure head motions in Euler angles of rotation around x-(pitch), y-(yaw), and z-axis (roll) respectively [5,18,34]. The three rotation angles are also defined in the MPEG-4 facial animation parameters (FAP) [32].…”

“…Early analysis has found that the timing of head motion accompanying speech is typically well synchronized with prosodic structure of the text [18]. An automatic textdriven head motion synthesizer was implemented by mapping the grammatical and prosodic textual information to elementary head motion patterns based on classification and regression trees (CART) [40].…”

The timing of head movements: The role of prosodic heads and edges

“…These studies take the position of the accented syllables as the key prosodic landmark with which gesture movements align, but they do not take into account intonational phrase boundaries. In general, they found a similar temporal alignment pattern as had been shown for hand gestures: accented syllables are the anchoring point in speech for the most prominent part of a head movement, the gesture apex (defined as the specific point in time when the head changes its direction in the vertical or lateral movement) (Alexanderson et al, 2013;Ambrazaitis et al, 2015;Fernandez-Baena et al, 2014;Goldenberg et al, 2014;Graf et al, 2002;Hadar et al, 1983;Ishi et al, 2014;Kim et al, 2014). However, these studies also reported variability in this alignment pattern.…”