Acoustic specification of upper limb movement in voicing: Exploratory data report and pre-registration

Abstract

<p>Hand gestures communicate complex information to listeners through the visual information created by movement. In a recent study, however, we found that there are also direct biomechanical effects of high-impetus upper limb movement on voice acoustics. Here we explored whether listeners could detect information about movement in voice acoustics of another person. In this exploratory study, participants listened to a recording of a vocalizer who was simultaneously producing low- (wrist movement) or high- (arm movement) impetus movements at three different tempos. Listeners were asked to synchronize their own movement (wrist or arm movement) with that of the vocalizer. Listeners coupled with the frequency of the vocalizer arm (but not wrist) movements, and showed phase-coupling with vocalizer arm (but not wrist) movements. However, we found that this synchronization occurred regardless of whether the listener was moving their wrist or arm. This study shows that, in principle, there is acoustic specification of arm movements in voicing, but not wrist movements. These results, if replicated, provide novel insight into the possible interpersonal functions of gesture acoustics, which may lie in communicating bodily states. The second part of the paper is a pre-registration for the confirmatory study that will assess the research question in a larger sample with more diverse and naturalistic stimuli.</p>

Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures

“…When moving the upper limbs so that the physical impulse of that movement cooccurs near the vocalization (in-phase condition) higher amplitude envelope and more respiration-related activity is found as compared to a passive (no-movement) condition, or as compared 90° out-of-phase movement condition. In all these cases, the heightened acoustic and respiration-related activity effects of the in-phase condition were more pronounced for arm movements as compared to wrist movements, replicating earlier work on more extreme physical impulses on vocalization acoustics when mass of the effector in motion is higher Pouw, Paxton, et al, 2019, 2020. There is a crucial role of physical impulse in the in-phase condition as vocalizations that occurred closer to the peak in deceleration had higher amplitude and concomitant respiration-related activity as compared to vocalizations that were further temporally removed from the peak physical impulse.…”

“…Although there is evidence suggesting that gestures affect acoustics via the respiratory system, and that respiration-related muscle activity is sometimes implicated in emphatically stressed speech, it is yet to be directly tested that gesturing and prosodic aspects of speech are linked via the respiration system. Additionally, direct evidence for gesture's biomechanical effects on acoustics are still based on a rather rudimentary paradigm Pouw, Paxton, et al, 2019, 2020 where subjects need to produce steady-state vocalizations. Although there is indirect evidence that gestures might indeed affect spontaneous speech through gesture-speech physics (e.g., Cravotta, Grazia, & Prieto, 2019), direct evidence is needed to show that physical impulse can affect more speech-like productions through respiratory modulations.…”