Neural correlates of the processing of co-speech gestures

Abstract

In communicative situations, speech is often accompanied by gestures. For example, speakers tend to illustrate certain contents of speech by means of iconic gestures which are hand movements that bear a formal relationship to the contents of speech. The meaning of an iconic gesture is determined both by its form as well as the speech context in which it is performed.

A common functional neural network for overt production of speech and gesture

“…The language network, together with inferior parietal cortex, is also engaged during sign language production (Emmorey et al, 2007;MacSweeney et al, 2008). Importantly, the first whole-brain pattern shows fronto-temporal areas that have repeatedly been found in studies of co-speech gesture perception, such as left inferior frontal gyrus and left superior temporal gyrus (Dick et al, 2012;Green et al, 2009;Holle et al, 2008Holle et al, , 2010Willems et al, 2007Willems et al, , 2009). In addition, these areas are associated with semantic processes, such as word retrieval (left inferior frontal gyrus: Amunts et al, 2004;de Zubicaray & McMahon, 2009;Thompson-Schill et al, 1999;Tremblay & Gracco, 2006; superior temporal gyrus: Graves et al, 2010;Heath et al, 2012;Khader et al, 2010).…”

“…It seems reasonable that cospeech gesture production engages the gesture network because it combines conceptual, as well as skill-related aspects of actions that are essential for the meaningful hand movements of co-speech gesturing. Importantly, the second activity pattern includes fronto-parietal areas that are also frequently found in neuroimaging studies of co-speech gesture perception (Dick et al, 2009;Holle et al, 2008;Kircher et al, 2009;Straube et al, 2011;Willems et al, 2007Willems et al, , 2009). …”

“…Their coordination with speech is both temporal-kinetic and semantic (Kita & Özyürek, 2003;Loehr, 2007) and they have been shown to affect learning and memory in the listener as well as the speaker (Goldin-Meadow, 2003;Hostetter, 2011;Marstaller & Burianová, 2013). Previous neuroimaging studies have found that the observation of co-speech gestures engages superior and middle temporal gyrus, intraparietal sulcus, and inferior frontal gyrus (Dick et al, 2009;Holle et al, 2008Holle et al, , 2010Hubbard et al, 2009;Kircher et al, 2009;Skipper et al, 2007Skipper et al, , 2009Straube et al, 2011;Willems et al, 2007Willems et al, , 2009). The findings from these studies strongly suggest that during the observation of co-speech gestures, frontal and temporal regions are engaged in semantic processing, whereas frontal and parietal areas are activated for action understanding (Marstaller & Burianová, 2014).…”

Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions

“…Or could the posterior superior temporal cortex possibly serve as a general audiovisual integration module (Beauchamp, Lee et al, 2004), which is capable both of lower form-level as well as higher semantic-conceptual integration processes? Unfortunately, the five published studies (Dick et al, in press;Green et al, in press;Holle et al, 2008;Willems et al, 2007Willems et al, , 2009 come to different conclusions: some studies argue that integration of iconic gestures and speech takes place in the IFG (Dick et al, in press;Willems et al, 2007Willems et al, , 2009, while others emphasize the involvement of posterior temporal areas (Green et al, in press;Holle et al, 2008). In some of these studies, alternative explanations unrelated to integration cannot be excluded.…”

“…In some of these studies, alternative explanations unrelated to integration cannot be excluded. One concern is that some studies could not demonstrate bimodal enhancement because they either lacked the necessary unimodal conditions (Dick et al, in press;Holle et al, 2008;Willems et al, 2007) or demonstrated bimodal enhancement only for a selected region of interest (Green et al, in press). …”