Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures

Abstract

Expressive moments in communicative hand gesture often align with emphatic stress in speech. It has recently been found that acoustic markers of emphatic stress arise naturally during steady-state phonation when upper-limb movements impart physical impulse on the body, most likely affecting acoustics via respiratory activity. In this confirmatory study, participants (N = 29) repeatedly uttered consonant-vowel CV (/pa/) mono-syllables while moving in particular phase relations with speech, or not moving the upper limbs. We show that respiration-related activity is affected by (especially high-impulse) gesturing when vocalizations occur near peaks in physical impulse. We further show that gesture-induced moments of bodily impulses increase the amplitude envelope of speech, while not similarly affecting the Fundamental Frequency (F0). Finally, tight relations between respirationrelated activity and vocalization were observed, even in the absence of movement, but even more so when upper-limb movement is present. The current findings expand a developing line of research showing that speech is modulated by functional biomechanical linkages between hand gesture and the respiratory system. This identification gesture-speech biomechanics promises to provide an alternative phylogenetic, ontogenetic, and mechanistic explanatory route of why communicative upper limb movements co-occur with speech in humans.

A Systematic Investigation of Gesture Kinematics in Evolving Manual Languages in the Lab

“…Indeed, in a study by Perlman, Dale, and Lupyan (2015), it is shown how dynamic aspects of vocalization signaling systems become more efficient, similar to our current reduction in kinematic complexity. These findings, together with work showing the tight connection between speech and gesture (Bosker & Peeters, 2020;Pouw, de Jonge-Hoekstra, Harrison, Paxton, & Dixon, 2020;Pouw, Harrison, Esteve-Gibert, & Dixon, 2020), make it a natural next step to look at multimodal iterated learning experiments. Furthermore, our approach can inform work on communicative alignment in conversations (Rasenberg, Özyürek, & Dingemanse, 2020) or the ways in which people can repeat aspects of each other's communicative behavior.…”

Multilevel rhythms in multimodal communication

“…This coupling of gestures' acceleration-induced forces and speech can arise biomechanically from upper limb-respiratory coupling, e.g. by soliciting anticipatory muscle adjustments to stabilize posture during gesture [52], which also include respiratory-controlling muscles supporting speech-vocalization [53]. Comparable biomechanical interactions and synergies have been found in other animals long before such associations were raised to explain aspects of human multimodal prosody.…”

“…Human communicative hand gestures have acceleration peaks co-occurrent with emphatic stress in speech, which are tightly and dynamically coupled under adverse conditions, though with more temporal variability for more complex symbolizing gestures [52]. This coupling of gestures' acceleration-induced forces and speech can arise biomechanically from upper limb-respiratory coupling, e.g., by soliciting anticipatory muscle adjustments to stabilize posture during gesture [53], which also include respiratorycontrolling muscles supporting speech-vocalization [54]. Comparable biomechanical interactions and synergies have been found in other animals long before such associations were raised to explain aspects of human multimodal prosody.…”

Probing effects of lexical prosody on speech-gesture integration in prominence production by Swedish news presenters

“…This alignment has, furthermore, shown to be affected by linguistic prosodic structure (e.g., Esteve-Gibert & Prieto, 2013;Krivokapić, 2014; Esteve-Gibert, Borràs-Comes, Asor, Swerts, & Prieto, 2017), making an even stronger case for speech and gesture being planned in conjunction than the mere co-occurrence. Moreover, speech and gesture not only converge in the temporal domain, but also in the 'spatial' domain in a broad sense, displaying correlations between the presence, magnitude, or complexity of movements with the extension of articulatory or acoustic parameters of speech (e.g., Krahmer & Swerts, 2007;Parrell, Goldstein, Lee, & Byrd, 2014;Pouw, Harrison, & Dixon, 2020a;Pouw, Harrison, Esteve-Gibert, & Dixon, 2020b;Pouw, de Jonge-Hoekstra, Harrison, Paxton, & Dixon, 2021).…”