The physical basis of gesture-speech synchrony: Exploratory study and pre-registration

Abstract

Hand gestures during speech move in a common rhythm, as exemplified by the synchrony between prosodic contrasts in gesture movement (e.g., peak velocity) and speech (e.g., peaks in Fundamental Frequency). This joined rhythmic activity is hypothesized to have a variable set of functions, ranging from self-serving cognitive benefits for the gesturer, to communicational advantages that support listeners' understanding. However, gesturespeech synchrony has been invariably understood as a "neural-cognitive" achievement; i.e., gesture and speech are coupled through neural-cognitive mediation. Yet, it is possible that gesture-speech synchrony emerges out of resonating forces that travel through a common physical medium -the body. The current paper presents an exploratory study together with a pre-registration of a larger scale confirmatory study. We provide preliminary evidence that upper limb motions with greater momentum (i.e., physical impetus) affects Fundamental Frequency and the Amplitude Envelope of phonation in a way that accommodates research on gesture-speech synchrony. We speculate that anticipatory postural adjustments and related physical effects of upper limb movements on the musculoskeletal connective system could lead to changes in alveolar (lung) pressure that can impart (prosodic) contrasts in phonation. Here we pre-registered a confirmatory study to more comprehensively address this hypothesis.

Acoustic information about upper limb movement in voicing

“…movements, however, acoustic modulations are absent (21). Similar to previous research (21,22), in the current study hand movements inadvertently affected voice acoustics of these prestudy vocalizer participants ( Fig. 1D) , thereby providing a possible information source for listeners in the main study.…”

“…It is important to note that vocalizers were thus not exposed to an external rhythmic signal, such as a (visual) metronome. Further note that we found in an earlier study that when vocalizers move at their own preferred tempo-with no visual feedback about movement tempo-acoustic modulations are also obtained that are tightly synchronized with movement cycles (22). If participants vocalize without…”

Gesture–speech physics: The biomechanical basis for the emergence of gesture–speech synchrony.

“…With the current pre-registered study (see https://osf.io/5aydk/), we aim to replicate and extend earlier exploratory findings (Pouw, Harrison, & Dixon, 2018) which indicated that upper limb movements with high physical impetus (one-arm beat and twoarm beat) were synchronized with peaks in F0 and the amplitude envelope during a RUNNING HEAD: Gesture-speech physics phonation task 1 . Here we investigate the role of physical impetus on phonation with a larger sample (10 participants; 240 trials) and assess the possible modulatory role of postural stability.…”

Acoustic specification of upper limb movement in voicing: Exploratory data report and pre-registration

“…Note. Example time series of the exploratory study reported in (Pouw, Harrison, & Dixon, 2018a;Pouw, Harrison, et al, 2018b) wherein a female vocalizer phonates the vowel /a:/ while moving her hand up and down with a wrist or arm motion or while standing passively. The black line shows the movement time series on the vertical dimension, the red line shows the concurrent Fundamental Frequency of the phonation, and the purple line shows the amplitude envelope, which is a measure of the energy of the voicing signal.…”

“…To lay the groundwork for our preregistration plan of a confirmatory study, we first report the results from an exploratory study in which two participants were asked to make a wrist or arm motion while listening to a recording from a vocalizer, an original participant from Pouw, Harrison, et al (2018a). The vocalizer was continuously voicing the vowel /a:/ while making a high-impetus arm motion or a low-impetus wrist motion at slow, medium, or fast movement tempos.…”