It is commonly understood that hand gesture and speech coordination in humans is culturally and cognitively acquired, rather than having a biological basis. Recently, however, the biomechanical physical coupling of arm movements to speech vocalization has been studied in steady-state vocalization and monosyllabic utterances, where forces produced during gesturing are transferred onto the tensioned body, leading to changes in respiratory-related activity and thereby affecting vocalization F0 and intensity. In the current experiment (n = 37), we extend this previous line of work to show that gesture-speech physics also impacts fluent speech. Compared with nonmovement, participants who are producing fluent self-formulated speech while rhythmically moving their limbs demonstrate heightened F0 and amplitude envelope, and such effects are more pronounced for higher-impulse arm versus lowerimpulse wrist movement. We replicate that acoustic peaks arise especially during moments of peak impulse (i.e., the beat) of the movement, namely around deceleration phases of the movement. Finally, higher deceleration rates of higher-mass arm movements were related to higher peaks in acoustics. These results confirm a role for physical impulses of gesture affecting the speech system. We discuss the implications of gesture-speech physics for understanding of the emergence of communicative gesture, both ontogenetically and phylogenetically.

Cross-recurrence quantification analysis (CRQA) is a powerful nonlinear time-series method to study coordination and cooperation between people. This chapter concentrates on two methodological issues related to CRQA on categorical data streams, which are commonly encountered in the behavioral sciences. Firstly, we introduce a more general definition of recurrence as 'behavioral matching', which can be applied to several kinds of matches simultaneously, visualized by a color coding. We will refer to this as cross-matching, and to the resulting quantification procedure as Chromatic CRQA. Secondly, cross-recurrence plots of categorical data often prominently consists of rectangular structures. This calls for a differential analysis of vertical and horizontal lines, rather than of diagonal lines. We introduce a simple procedure for this, referred to as Anisotropic CRQA. Both procedures are demonstrated with empirical studies on children's problem-solving behavior and by means of a model simulation. The authors hope that the ideas presented here increase the power and applicability of CRQA in the behavioral sciences, and that this chapter serves as a stepping stone for their mathematical and methodological development.

As children learn they use their speech to express words and their hands to gesture. This study investigates the interplay between real-time gestures and speech as children construct cognitive understanding during a hands-on science task. 12 children (M = 6, F = 6) from Kindergarten (n = 5) and first grade (n = 7) participated in this study. Each verbal utterance and gesture during the task were coded, on a complexity scale derived from dynamic skill theory. To explore the interplay between speech and gestures, we applied a cross recurrence quantification analysis (CRQA) to the two coupled time series of the skill levels of verbalizations and gestures. The analysis focused on (1) the temporal relation between gestures and speech, (2) the relative strength and direction of the interaction between gestures and speech, (3) the relative strength and direction between gestures and speech for different levels of understanding, and (4) relations between CRQA measures and other child characteristics. The results show that older and younger children differ in the (temporal) asymmetry in the gestures–speech interaction. For younger children, the balance leans more toward gestures leading speech in time, while the balance leans more toward speech leading gestures for older children. Secondly, at the group level, speech attracts gestures in a more dynamically stable fashion than vice versa, and this asymmetry in gestures and speech extends to lower and higher understanding levels. Yet, for older children, the mutual coupling between gestures and speech is more dynamically stable regarding the higher understanding levels. Gestures and speech are more synchronized in time as children are older. A higher score on schools’ language tests is related to speech attracting gestures more rigidly and more asymmetry between gestures and speech, only for the less difficult understanding levels. A higher score on math or past science tasks is related to less asymmetry between gestures and speech. The picture that emerges from our analyses suggests that the relation between gestures, speech and cognition is more complex than previously thought. We suggest that temporal differences and asymmetry in influence between gestures and speech arise from simultaneous coordination of synergies.

No abstract

Hand-gestures are seamlessly coordinated with speech. Yet, there is only anecdotal support for gestures’ functional role in speech production. Here we explore temporal aspects of speech production when people use hand gesture. We performed exploratory analyses with a naturalistic German-speaking sample from The Bielefeld Speech and Gesture Alignment Corpus (SaGA), which consisted of 67 minutes of narration data and over 500 gesture events (N = 6). We found that the rhythmic timing of speech (defined as the mean and standard deviations of speech onset intervals) is highly correlated with the likelihood of gesturing. Furthermore, we utilized deep learning methods to track gesture motion, and extracted the amplitude envelope of speech, so as to gauge the degree of (continuous) gesture-speech synchrony. We then performed a continuous time-series analysis (recurrence quantification analysis; RQA) to index how temporal properties of speech change when gesture and speech are more or less synchronized. Our analyses revealed that when gesture and speech were more synchronized, the temporal structure of speech was more ordered and less complex, as indexed by classic measures of dynamic temporal stability (e.g., Entropy, Ratio of %Determinism/Recurrence). We suggest that a fundamental gesture-speech relation is rooted in entrainment, which yields stability in the temporal structure of speech.

Gestures and speech are clearly synchronized in many ways. However, previous studies have shown that the semantic similarity between gestures and speech breaks down as people approach transitions in understanding. Explanations for these gesture–speech mismatches , which focus on gestures and speech expressing different cognitive strategies, have been criticized for disregarding gestures’ and speech's integration and synchronization. In the current study, we applied three different perspectives to investigate gesture–speech synchronization in an easy and a difficult task: temporal alignment , semantic similarity , and complexity matching . Participants engaged in a simple cognitive task and were assigned to either an easy or a difficult condition. We automatically measured pointing gestures, and we coded participant's speech, to determine the temporal alignment and semantic similarity between gestures and speech. Multifractal detrended fluctuation analysis was used to determine the extent of complexity matching between gestures and speech. We found that task difficulty indeed influenced gesture–speech synchronization in all three domains. We thereby extended the phenomenon of gesture–speech mismatches to difficult tasks in general. Furthermore, we investigated how temporal alignment, semantic similarity, and complexity matching were related in each condition, and how they predicted participants’ task performance. Our study illustrates how combining multiple perspectives, originating from different research areas (i.e., coordination dynamics, complexity science, cognitive psychology), provides novel understanding about cognitive concepts in general and about gesture–speech synchronization and task difficulty in particular.

No abstract