A Kinematic Study of Prosodic Structure in Articulatory and Manual Gestures: Results from a Novel Method of Data Collection

Abstract

The primary goal of this work is to examine prosodic structure as expressed concurrently through articulatory and manual gestures. Specifically, we investigated the effects of phrase-level prominence (Experiment 1) and of prosodic boundaries (Experiments 2 and 3) on the kinematic properties of oral constriction and manual gestures. The hypothesis guiding this work is that prosodic structure will be similarly expressed in both modalities. To test this, we have developed a novel method of data collection that simultaneously records speech audio, vocal tract gestures (using electromagnetic articulometry) and manual gestures (using motion capture). This method allows us, for the first time, to investigate kinematic properties of body movement and vocal tract gestures simultaneously, which in turn allows us to examine the relationship between speech and body gestures with great precision. A second goal of the paper is thus to establish the validity of this method. Results from two speakers show that manual and oral gestures lengthen under prominence and at prosodic boundaries, indicating that the effects of prosodic structure extend beyond the vocal tract to include body movement.1

The physical basis of gesture-speech synchrony: Exploratory study and pre-registration

“…Despite having been largely disregarded by phoneticians and gesture researchers, we think that there is a viable possibility that upper limb movements in gesturing have direct physical effects on F0 and amplitude which, perhaps in part, provide a means for gesture-speech synchronization as observed in spontaneous gesture-speech synchrony (e.g., Krivokapić et al, 2017;Wagner et al, 2014). Similar to weakly coupled oscillators that spontaneously synchronize due to vibrations traveling through a shared physical medium (e.g., pressure waves; shared physical platform; see Pikovsky, Rosenblum, Kurths, 2001), the body allows -and is dependent in its functioning on -forces that resonate through its musco-skeletal network (Turvey & Fonseca, 2014).…”

“…Hand gesture and speech are closely synchronized (see for an example https://osf.io/29h8z/; Leonard & Cummins, 2010;Krivokapić, Tiede, Tyrone, & Goldenberg, 2016;Krivokapić, Tiede, Tyrone, 2017;Pouw & Dixon, 2018a, b; Parrel, Goldstein, Lee, & Byrd, 2014;Rochet-Capellan, Shaiman, Iverson, & Szumisky, 2014;Treffner & Peter, 2002;Zelic, Kim, & Davis, 2015). Specifically, speech' prosodic contrasts, captured by contrasts in the fundamental frequency of speech (F0; perceived as the 'pitch' of speech), structurally aligns with energetic contrasts in gesture (e.g., peak velocity or point of maximum effort; Krivokapić, Tiede, & Tyrone, Goldenberg, 2016;Loehr, 2004;Pouw & Dixon, 2018a, 2018b.…”

The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

“…Timing is even more finely tuned on the prosodic level, where it has been found that gesture-speech coupling occurs on even shorter timescales (200 ms). For example, it has been found, in the pronunciation of the nonsense word "baba," when stress is put on the first (baba) or last syllable (baba), the pointing gesture's maximum extension is coordinated to align more closely with the stressed syllable (Rochet-Capellan, Laboissiere, Galvan, & Schwartz, 2008; see also Esteve-Gibert & Prieto, 2013;Krivokapić, Tiede, & Tyrone, 2017;Rusiewicz, Shaiman, Iverson, & Szuminsky, 2014). On the biomechanical level, it has further been found that gesture's moments of peak physical impetus (lasting~50 ms) entrain fundamental frequency and the amplitude envelope during phonation (Pouw, Harrison, & Dixon, 2019).…”

“…It is clear, therefore, that if the laborious task of hand coding gesture kinematics can be replaced with reliable and objective automated methods, considerable advantages will accrue for researchers. In the present article, we contribute to this joint effort by introducing and validating novel methods for studying gesture-speech dynamics (e.g., Beecks et al, 2015;Danner et al, 2018;Krivokapić et al, 2017;Schueller et al, 2017;Pouw & Dixon 2019b), and by providing a tutorial for overcoming some common challenges when doing multimodal research.…”

Research methods in articulatory phoneticsII: Studying other gestures and recent trends

“…For greater temporal resolution (e.g., for studying signing rate effects, for example, Tyrone & Mauk, 2012; or coordination with lingual gestures, for example, Krivokapic, Tiede, & Tyrone, 2017), optical tracking is a preferred tool. The latter method also provides more accurate spatial positions of relevant gestures.…”