Visualizing prosodic structure: Manual gestures as highlighters of prosodic heads and edges in English academic discourses

Multimodal prosody: gestures and speech in the perception of prominence in Spanish

“…This suggests that, despite the large effect size observed for holds, when the number of marks given in the audiovisual modality was compared to those marks given in the audio-only modality, it was mostly strokes-and to a lesser extent also apexes-that made participants give words a mark of prominence. Strokes typically coincide with stressed syllables (e.g., Loehr, 2012;Esteve-Gibert and Prieto, 2013;Rohrer et al, 2023) and might be perceptually more salient than apexes, which correspond to the part of strokes that is time-aligned with F0 peaks. In addition, by the size of their effect, the visual cues of prominence seem to have contributed more than the auditory cues to drive the perception of participants.…”

“…Furthermore, the preponderant role of duration as a correlate both of lexical stress and phrasal stress in Spanish has also been attested (Vogel et al, 2016). As for the interaction between speech and gestures, there exists a strong temporal coordination between stressed syllables and strokes (McNeill, 1992), and more specifically between F0 peaks and the apex of gestures, i.e., the effort peak within strokes (Kendon, 1972;Loehr, 2004Loehr, , 2012Renwick et al, 2004;Jannedy and Mendoza-Denton, 2005;Shattuck-Hufnagel et al, 2007;Esteve-Gibert and Prieto, 2013;Rohrer et al, 2023).…”

“…Nonetheless, not only the external validity of both methods could be improved, but also their potential to study the complex interaction of visual and verbal prosody is to some extent reduced, especially since most studies have focused on the interaction of just one gesture articulator-either hands, head, or eyebrows-with one acoustic correlate of prominence, which is very far from the natural interweaving of gestures performed with hands, head, and eyebrows typically found in everyday spoken language. Alternatively, rather than relying solely on animated agents and multimodal stimuli elicited in experimental settings, researchers have also used spontaneous speech and spontaneously elicited gestures in order to gain new insights into the interaction between visual and verbal prosody (Swerts and Krahmer, 2010;Ambrazaitis and House, 2017;Jiménez-Bravo and Marrero-Aguiar, 2020;Rohrer et al, 2023). One of the first studies using spontaneous speech analyzed the effects of facial expressions on speech production using samples obtained from two male and two female Dutch TV newsreaders (Swerts and Krahmer, 2010).…”

“…Similarly, the connection between prosody and gestures using multimodal stimuli was also addressed by Rohrer et al (2023), who analyzed a corpus of academic discourses in English (TED Talks). The researchers observed that the patterns of temporal alignment of strokes and apexes within the boundaries of pitch accented syllables was different for both gesture phases, with strokes spanning over pitch accented syllables more frequently than apexes.…”