Easier said than done? Task difficulty’s influence on temporal alignment, semantic similarity, and complexity matching between gestures and speech

Abstract

Gestures and speech are clearly synchronized in many ways. However, previous studies have shown the semantic similarity between gestures and speech breaks down as people approach transitions in understanding. Explanations for these gesture-speech mismatches which focus on gestures and speech expressing different cognitive strategies, have been criticized for disregarding gestures’ and speech’s integration and synchronization. In the current study, we applied three different perspectives to investigate gesture-speech synchronization in an easy and a difficult task: temporal alignment, semantic similarity, and complexity matching. Participants engaged in a simple cognitive task, and were assigned to either an easy or a difficult condition. We automatically measured pointing gestures, and we coded participant’s speech, to determine the temporal alignment and semantic similarity between gestures and speech. Multifractal Detrended Fluctuation Analysis (MFDFA) was used to determine the extent of complexity matching between gestures and speech. We found that task difficulty indeed influenced gesture-speech synchronization in all three domains. We thereby extended the phenomenon of gesture-speech mismatches to difficult tasks in general. Furthermore, we investigated how temporal alignment, semantic similarity, and complexity matching were related in each condition, and how they predicted participants’ task performance. Our study illustrates how combining multiple perspectives, originating from different research areas (i.e., coordination dynamics, complexity science, cognitive psychology) provides novel understanding about cognitive concepts in general, and about gesture-speech synchronization and task difficulty in specific.

Gesture-speech physics in fluent speech and rhythmic upper limb movements

“…This is similar to research showing rhythmic stabilities arising out upper limb movement and their interactions with respiration cycles (e.g., 25,26 ). Thus, the upper limb and speech system naturally couples its activity, like many other living as well as non-living oscillatory systems ( 27 ; also see 28 ), requiring further study on the exact nature of this coupling.…”

Multifractal Mice: Measuring Task Engagement and Readiness-to-hand via Hand Movement

“…WTMM takes a one dimensional data input, and in MFA experiments it is common to analyse a single dimension of activity, and even manipulating data into this format where necessary (e.g. [4,15,16,23,56]). This practice follows from Takens' theorem, which demonstrates that multifractal dynamics in a system can be reconstructed from measurement of any dimension in that system [41].…”

“…Multifractality is a measure of system complexity, developed in fluid mechanics, but now important in a range of fields [47], including physiology [6,34,55,75,83] and cognitive science [32,44,52,76]. Multifractality indicates the role of complex nonlinear interactions in a system's behaviour [42,52], and a large literature now demonstrates its presence in a wide range of human behaviours [15,16,23,33,35,46,57,65,76,77,84], often associated with adaptive flexibility [4,56,67].…”

“…The growing importance of multifractality as an analytical tool has led to a number of tutorials and primers on the topic focusing on human behaviour [3,16,23,28,42,43,52,54], as well as a number of libraries and tools implementing analysis techniques [38,48,73]. Due to the wide availability of these materials, we do not aim to contribute another tutorial, or a detailed account of the mathematical foundations -such papers exceed the length of this one, even without discussing experimental results.…”

“…Clear introductions to WTMM can be found in [3,23] and to MDFA in [16,43], the latter of which gives a walk through of the analysis in MATLAB. More detailed accounts of each are also available -either more mathematical [44, 49?…”