Movers and shakers of cognition: Hand movements, speech, task properties, and variability

Abstract

Children move their hands to explore, learn and communicate about hands-on tasks. Their hand movements seem to be “learning” ahead of speech. Children shape their hand movements in accordance with spatial and temporal task properties, such as when they feel an object or simulate it’s movements. Their speech does not directly correspond to these spatial and temporal task properties, however. We aimed to understand whether and how hand movements’ are leading cognitive development due to their ability to correspond to spatiotemporal task properties, while speech is unable to do so. We explored whether hand movements’ and speech’s variability changed with a change in spatiotemporal task properties, using two variability measures: Diversity indicates adaptation, while Complexity indicates flexibility. In two experiments, we asked children (4-7 years) to predict and explain about balance scale problems, whereby we either manipulated the length of the balance scale or the mass of the weights after half of the trials. In three out of four conditions, we found a change in Complexity for both hand movements and speech between first and second half of the task. In one of these conditions, we found a relation between the differences in Complexity and Diversity of hand movements and speech. Changes in spatiotemporal task properties thus often influenced both hand movements’ and speech’s flexibility, but there seem to be differences in how they did so. We provided many directions for future research, to further unravel the relations between hand movements, speech, task properties, variability, and cognitive development.

Easier said than done? Task difficulty’s influence on temporal alignment, semantic similarity, and complexity matching between gestures and speech

“…First, both in the current study and in previous studies, gestures had a clear spatial component that was directly linked to the physical properties of the task material (e.g. Bergmann & Kopp, 2010;de Jonge-Hoekstra et al, 2019;Hostetter & Alibali, 2008;Yeo & Alibali, 2018). This is not true for speech, however, and Smith and Gasser (2005) even propose that a too close resemblance between the physical structure of the environment and the structure of speech would limit speech's functionality.…”

Easier Said Than Done? Task Difficulty's Influence on Temporal Alignment, Semantic Similarity, and Complexity Matching Between Gestures and Speech

“…We aimed to investigate how task difficulty affects the synchronization between gestures and speech, thereby empirically addressing De Jonge‐Hoekstra et al.’s (2016) proposal. By doing so, we brought together different perspectives and ways of investigating gesture–speech synchronization.…”

“…This is in line with the positive relation between temporal alignment and semantic similarity and less temporal alignment, and also with less temporal alignment in the difficult condition, that we found. Follow‐up studies could research the relation between gestures, speech, and gesture–speech mismatches in more detail, using methods to quantify the temporal direction of gesture–speech coupling, such as cross recurrence quantification analysis (see also De Jonge‐Hoekstra et al., 2016). Moreover, in previous studies, temporal information usually has been disregarded when coding gesture–speech mismatches (e.g., Alibali et al., 2000).…”

“…As put forward by De Jonge‐Hoekstra et al. (2020), a metaphor for this reorganization is building a new LEGO‐structure from an old structure, which is only possible when you break (perturb) the old structure and use the bricks to create a new structure. Taking such a dynamically embodied, complex system's perspective, De Jonge‐Hoekstra, Van der Steen, Van Geert, and Cox (2016) suggest that difficult tasks perturb a system, thereby inducing a suboptimal coordination between gestures and speech, which could then lead to various forms of gesture–speech mismatches.…”

How children make sense of the world: A perceptual learning account

“…Third, this operationalization of perceptual sensitivity has considerable conceptual overlap with the notion of 'differentiation' as put forward by Gibson and Gibson (1955), as the sensitivity for perceiving distinctions leading to differential responses plays a central role in both. Finally, Shannon's Diversity Index (see Begon, Harper, & Townsend, 1996;de Jonge-Hoekstra, van der Steen, & Cox, 2020) was used to index the diversity of incorrect responses (i.e., kinds of false alarms) across age groups.…”