Gesture–speech physics in fluent speech and rhythmic upper limb movements

Abstract

It is commonly understood that hand gesture and speech coordination in humans is culturally and cognitively acquired, rather than having a biological basis. Recently, however, the biomechanical physical coupling of arm movements to speech vocalization has been studied in steady-state vocalization and monosyllabic utterances, where forces produced during gesturing are transferred onto the tensioned body, leading to changes in respiratory-related activity and thereby affecting vocalization F0 and intensity. In the current experiment (n = 37), we extend this previous line of work to show that gesture-speech physics also impacts fluent speech. Compared with nonmovement, participants who are producing fluent self-formulated speech while rhythmically moving their limbs demonstrate heightened F0 and amplitude envelope, and such effects are more pronounced for higher-impulse arm versus lowerimpulse wrist movement. We replicate that acoustic peaks arise especially during moments of peak impulse (i.e., the beat) of the movement, namely around deceleration phases of the movement. Finally, higher deceleration rates of higher-mass arm movements were related to higher peaks in acoustics. These results confirm a role for physical impulses of gesture affecting the speech system. We discuss the implications of gesture-speech physics for understanding of the emergence of communicative gesture, both ontogenetically and phylogenetically.

A kinematic-acoustic analysis of gesture-speech coupling in persons with aphasia

“…Namely, we perform several analyses that provide fully reproducible procedures to probe whether speech and gesture activity are comodulated, whether they synchronize in time, and how gesture and speech rhythms are related. This approach is comparable to recent methods that have been reported in several other studies (Pouw, de Jonge-Hoekstra, Harrison, et al, 2020;Pouw, Trujillo, & Dixon, 2020;Pouw & Dixon, 2019a, 2019b. The following research questions will be answered:…”

“…That is, acoustic-prosodic modulations in speech and vocalizations "timed" with gesture kinematic contrasts (e.g., "beating" moments in gestures) are partially instantiated by interactions between mechanical loading of the upper limbs onto the respiratory-vocal system. Especially kinematically contrastive upper limb movements, such as gestures with a beat quality, can produce forces onto the musculo-skeletal system that can affect the Fundamental Frequency (F0) and the amplitude of the voice through interactions with respiratory-vocal system Pouw, de Jonge-Hoekstra, Harrison, et al, 2020;Pouw, Harrison, Esteve-Gibert, et al, 2020, and F0 and the amplitude are key markers of prosodic contrasts. These experiments and several related findings (Cravotta et al, 2019;, provide support for the idea that there is a feedback of the gesture and speech trajectory that instantiate the gesture-speech formulation process partly in biomechanical processes.…”

“…Can we find magnitude coupling, such that peaks in changes in amplitude envelope are related to nearest peaks in acceleration profile of gesture? Acceleration and amplitude envelope coupling is a possible quantitative marker of gesture-speech biomechanics (Pouw, de Jonge-Hoekstra, Harrison, et al, 2020).…”

“…From these keypoint position x and y traces we calculated the 2D speed. In the analysis we also refer to acceleration which is the 1st derivative of speed as there is reason to believe that this kinematic feature is especially informative about gesture-speech dynamics (Ferstl et al, 2020;Pouw, de Jonge-Hoekstra, Harrison, et al, 2020). Acceleration was absolutized (so higher values can mean higher acceleration or deceleration).…”

A Systematic Investigation of Gesture Kinematics in Evolving Manual Languages in the Lab

“…Indeed, in a study by Perlman, Dale, and Lupyan (2015), it is shown how dynamic aspects of vocalization signaling systems become more efficient, similar to our current reduction in kinematic complexity. These findings, together with work showing the tight connection between speech and gesture (Bosker & Peeters, 2020;Pouw, de Jonge-Hoekstra, Harrison, Paxton, & Dixon, 2020;Pouw, Harrison, Esteve-Gibert, & Dixon, 2020), make it a natural next step to look at multimodal iterated learning experiments. Furthermore, our approach can inform work on communicative alignment in conversations (Rasenberg, Özyürek, & Dingemanse, 2020) or the ways in which people can repeat aspects of each other's communicative behavior.…”