Neural evidence for the interplay between language, gesture, and action: A review

Abstract

Co-speech gestures embody a form of manual action that is tightly coupled to the language system. As such, the co-occurrence of speech and co-speech gestures is an excellent example of the interplay between language and action. There are, however, other ways in which language and action can be thought of as closely related. In this paper we will give an overview of studies in cognitive neuroscience that examine the neural underpinnings of links between language and action. Topics include neurocognitive studies of motor representations of speech sounds, action-related language, sign language and co-speech gestures. It will be concluded that there is strong evidence on the interaction between speech and gestures in the brain. This interaction however shares general properties with other domains in which there is interplay between language and action.

A common functional neural network for overt production of speech and gesture

“…The language network, together with inferior parietal cortex, is also engaged during sign language production (Emmorey et al, 2007;MacSweeney et al, 2008). Importantly, the first whole-brain pattern shows fronto-temporal areas that have repeatedly been found in studies of co-speech gesture perception, such as left inferior frontal gyrus and left superior temporal gyrus (Dick et al, 2012;Green et al, 2009;Holle et al, 2008Holle et al, , 2010Willems et al, 2007Willems et al, , 2009). In addition, these areas are associated with semantic processes, such as word retrieval (left inferior frontal gyrus: Amunts et al, 2004;de Zubicaray & McMahon, 2009;Thompson-Schill et al, 1999;Tremblay & Gracco, 2006; superior temporal gyrus: Graves et al, 2010;Heath et al, 2012;Khader et al, 2010).…”

“…It seems reasonable that cospeech gesture production engages the gesture network because it combines conceptual, as well as skill-related aspects of actions that are essential for the meaningful hand movements of co-speech gesturing. Importantly, the second activity pattern includes fronto-parietal areas that are also frequently found in neuroimaging studies of co-speech gesture perception (Dick et al, 2009;Holle et al, 2008;Kircher et al, 2009;Straube et al, 2011;Willems et al, 2007Willems et al, , 2009). …”

“…Their coordination with speech is both temporal-kinetic and semantic (Kita & Özyürek, 2003;Loehr, 2007) and they have been shown to affect learning and memory in the listener as well as the speaker (Goldin-Meadow, 2003;Hostetter, 2011;Marstaller & Burianová, 2013). Previous neuroimaging studies have found that the observation of co-speech gestures engages superior and middle temporal gyrus, intraparietal sulcus, and inferior frontal gyrus (Dick et al, 2009;Holle et al, 2008Holle et al, , 2010Hubbard et al, 2009;Kircher et al, 2009;Skipper et al, 2007Skipper et al, , 2009Straube et al, 2011;Willems et al, 2007Willems et al, , 2009). The findings from these studies strongly suggest that during the observation of co-speech gestures, frontal and temporal regions are engaged in semantic processing, whereas frontal and parietal areas are activated for action understanding (Marstaller & Burianová, 2014).…”

Vision, action and language unified through embodiment

“…In the last decade this claim has been strongly supported by a series of experimental studies (for reviews see Pulvermüller, 2005;Willems & Hagoort, 2007). First, in an event-related fMRI study, the silent reading of words referring to face, arm or leg actions activated premotor-motor areas related to the word meanings (Hauk, Johnsrude, & Pulvermüller, 2004).…”

On the dynamics of action representations evoked by names of manipulable objects.

“…Language tasks that require the comprehension of action words (verbs like eat, walk, grasp) or nouns denoting manipulable objects (e.g., cup, pencil, cellphone) activate motor cortical regions, including the premotor and motor cortex (see Willems & Hagoort, 2007, for a concise review). This phenomenon, a kind of motor resonance, has been widely documented using a variety of different methods, including neuroimaging techniques and transcranial magnetic stimulation (TMS).…”