Hand gestures during speech move in a common rhythm, as exemplified by the synchrony between prosodic contrasts in gesture movement (e.g., peak velocity) and speech (e.g., peaks in Fundamental Frequency). This joined rhythmic activity is hypothesized to have a variable set of functions, ranging from self-serving cognitive benefits for the gesturer, to communicational advantages that support listeners' understanding. However, gesturespeech synchrony has been invariably understood as a "neural-cognitive" achievement; i.e., gesture and speech are coupled through neural-cognitive mediation. Yet, it is possible that gesture-speech synchrony emerges out of resonating forces that travel through a common physical medium -the body. The current paper presents an exploratory study together with a pre-registration of a larger scale confirmatory study. We provide preliminary evidence that upper limb motions with greater momentum (i.e., physical impetus) affects Fundamental Frequency and the Amplitude Envelope of phonation in a way that accommodates research on gesture-speech synchrony. We speculate that anticipatory postural adjustments and related physical effects of upper limb movements on the musculoskeletal connective system could lead to changes in alveolar (lung) pressure that can impart (prosodic) contrasts in phonation. Here we pre-registered a confirmatory study to more comprehensively address this hypothesis.
A well-known phenomenon of multimodal language is the synchronous coupling of prosodic contours in speech with salient kinematic changes in co-speech hand-gesture motions. Invariably, such coupling has been rendered by psychologists to require a dedicated neural-cognitive mechanism preplanning speech and gesture trajectories. Recently, in a continuous vocalization task, it was found that acoustic peaks unintentionally appear in vocalizations when gesture motions reach peaks in physical impetus, suggesting a biomechanical basis for gesture-speech synchrony (Pouw, Harrison, & Dixon, 2019). However, from this rudimentary study it is still difficult to draw strong conclusions about gesture-speech dynamics in (more) complex speech and the precise biomechanical nature of these effects. Here we assess how the timing of physical impetus of a gesture relates to its effect on acoustic parameters of mono-syllabic consonant-vowel (CV) vocalization(/pa/). Furthermore, we assess how chest-wall kinematics is affected by gesturing, and whether this modulates the effect of gestures on acoustics. In the current exploratory analysis, we analyze a subset (N = 4) of an already collected dataset (N = 36), which serves as the basis for a pre-registration of the confirmatory analyses yet to be completed. Here we provide exploratory evidence that gestures affect acoustics (amplitude envelope and F0) as well as chest-wall kinematics during mono-syllabic vocalizations. These effects are more extreme when a gesture’s peak impetus occurs closer to the center of the vowel vocalization event. If the current findings can be replicated in confirmatory fashion, there is a more compelling case to be made that gesture-speech physics is important facet of multimodal synchrony.
Hand gestures communicate complex information to listeners through the visual information created by movement. In a recent study, however, we found that there are also direct biomechanical effects of high-impetus upper limb movement on voice acoustics. Here we explored whether listeners could detect information about movement in voice acoustics of another person. In this exploratory study, participants listened to a recording of a vocalizer who was simultaneously producing low- (wrist movement) or high- (arm movement) impetus movements at three different tempos. Listeners were asked to synchronize their own movement (wrist or arm movement) with that of the vocalizer. Listeners coupled with the frequency of the vocalizer arm (but not wrist) movements, and showed phase-coupling with vocalizer arm (but not wrist) movements. However, we found that this synchronization occurred regardless of whether the listener was moving their wrist or arm. This study shows that, in principle, there is acoustic specification of arm movements in voicing, but not wrist movements. These results, if replicated, provide novel insight into the possible interpersonal functions of gesture acoustics, which may lie in communicating bodily states. The second part of the paper is a pre-registration for the confirmatory study that will assess the research question in a larger sample with more diverse and naturalistic stimuli.
We show that upper limb movement affects voice acoustics in a way that allows listeners to synchronize to those movements. In this pre-registered motion-tracking study (within-subjects, N = 30), participants listened to vocalizers producing a steady-state phonation while moving the upper limb with a wrist versus an arm motion at different tempos. Listeners were asked to synchronize their own wrist or arm movement with what they perceived in the voice acoustics to be the movement of the vocalizer. Previous research has shown that higher physical impetus gestures can directly affect lower vocal tract activity, leading to peaks in the fundamental frequency (F0; perceived as pitch) and intensity. Here we show that listeners can attune to acoustic information so as to synchronize both in frequency and in phasing with the vocalizer, even with very subtle movements. We interpret the current results in support of a possible ecological psychological approach to speech prosody.
In fluent speech, moments of acoustic prominence are tightly coordinated with peaks in the movement profile of hand gestures (e.g., speed of a gesture). This gesture-speech coordination has been found to operate on continuous bidirectional feedback of upper-limb movement. Here, we investigated the gesture-speech coordination of a person with deafferentation, the well-studied case of IW. Although IW has lost both his primary source of information about body position (i.e., proprioception) and touch, his gesture-speech coordination has been reported to be largely unaffected temporally and semantically, even if his vision is blocked. This is surprising as, without vision, his object-directed actions (e.g., grasping a cup) almost completely break down. Given differences in control in IW of gestures vs. actions when vision is unavailable, it has been suggested that communicative gesture operates under separate neural-cognitive constraints. In the current kinematic-acoustic study, we reanalyzed the classic 1998 and 2002 gesture experiments with IW (McNeill, 2005). Extending previous findings, we show that the micro-scale gesture-speech synchrony is compromised when vision is blocked, despite macro-scale coherence. Finally, we show a biomechanical linkage which could explain why IW’s gesture-speech capabilities in the absence of visual information is only mildly compromised.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.