Multimodal Prominence Marking in Semi-Spontaneous YouTube Monologs: The Interaction of Intonation and Eyebrow Movements

Abstract

Both facial expressions like eyebrow movements and prosodic characteristics like pitch height and the position of the pitch accent relative to the prominent syllable play an important role in prominence marking, which in turn is used by YouTubers and other public speakers alike to engage their audience. We analyze the co-occurrence of eyebrow movements and pitch accents of differing prominence levels produced by a sample of English-speaking YouTubers, and the height and position of those pitch accents. We annotated pitch accents, and analyzed videos using OpenFace 2.0, using three different eyebrow movement measures: eyebrow landmarks, distance between eyebrow and eye landmarks, and Action Units (facial expressions). We found that pitch height and eyebrow movements are positively correlated for at least some of the measures for all speakers. We also found that it is more likely for a medial or late pitch accent to arise at the time of an Action Unit connected to the eyebrows than an early accent, while there was no effect of the prominence level. However, we did not find evidence for a close temporal coordination of pitch movement and eyebrow movement. Finally, we found speaker-individual differences in the use of eyebrow movements and pitch accents, but also some slight indications of an effect for geographical origin of the speaker.

The Temporal Alignment of Speech-Accompanying Eyebrow Movement and Voice Pitch: A Study Based on Late Night Show Interviews

“…Unlike the other studies mentioned above, Berger and Zellers [68] used naturalistic data-semi-spontaneous YouTube monologs-to investigate the interaction of intonation and eyebrow movement. On the basis of automatic OpenFace annotations and manual pitch annotations, the authors found evidence of correlations between "pitch height and eyebrow movements .…”

“…. for at least some of the measures for all speakers" [68] (p. 1). Moreover, they went beyond previous studies aiming to "investigate the relationships between data in the form of contours rather than individual points" [68] (p. 10).…”

“…for at least some of the measures for all speakers" [68] (p. 1). Moreover, they went beyond previous studies aiming to "investigate the relationships between data in the form of contours rather than individual points" [68] (p. 10). For each pitch accent, they determined eyebrow trajectories around the F0-peak in a window of 1 s. Using Functional Principal Component Analysis, the authors did not find evidence for synchronization of eyebrow peaks and F0 peaks, contradicting the findings of Flecha-García [55].…”

“…The present study follows up on Berger and Zellers [68] in two respects. First, it uses naturalistic, rather than elicited data.…”

Multimodal marks of iteration in discourse

“…Likewise, no relation was found between eyebrow movement and OVER AND OVER. Measuring pitch movement may shed light on this result, given the interaction between eyebrow movement and intonational contours (e.g Berger & Zellers 2022)…”