Spatial–temporal dynamics of gesture–speech integration: a simultaneous EEG-fMRI study

Abstract

The semantic integration between gesture and speech (GSI) is mediated by the left posterior temporal sulcus/middle temporal gyrus (pSTS/MTG) and the left inferior frontal gyrus (IFG). Evidence from electroencephalography (EEG) suggests that oscillations in the alpha and beta bands may support processes at different stages of GSI. In the present study, we investigated the relationship between electrophysiological oscillations and blood-oxygen-level-dependent (BOLD) activity during GSI. In a simultaneous EEG-fMRI study, German participants (n = 19) were presented with videos of an actor either performing meaningful gestures in the context of a comprehensible German (GG) or incomprehensible Russian sentence (GR), or just speaking a German sentence (SG). EEG results revealed reduced alpha and beta power for the GG vs. SG conditions, while fMRI analyses showed BOLD increase in the left pSTS/MTG for GG > GR ∩ GG > SG. In time-window-based EEG-informed fMRI analyses, we further found a positive correlation between single-trial alpha power and BOLD signal in the left pSTS/MTG, the left IFG, and several sub-cortical regions. Moreover, the alpha-pSTS/MTG correlation was observed in an earlier time window in comparison to the alpha-IFG correlation, thus supporting a two-stage processing model of GSI. Our study shows that EEG-informed fMRI implies multiple roles of alpha oscillations during GSI, and that the method is a best candidate for multidimensional investigations on complex cognitive functions such as GSI.

The Role of Gamma Oscillations During Integration of Metaphoric Gestures and Abstract Speech

“…There are a number of explanations: firstly, even if these stimulus features provide information on both lower- and higher-order characteristics, none of these features directly address the critical word. This is however the position that directly relates to semantic integration between gesture and speech ( Obermeier and Gunter, 2014 ; He et al, 2018 ), and was analyzed to reveal the gamma-band effect. As a result, the obtained stimulus features were probably not sensitive enough to predict gamma-band power on a single-item basis.…”

“…The onset of the critical word is considered to be most important for the temporal synchronization between speech and gesture ( Habets et al, 2011 ; Obermeier et al, 2011 ; Obermeier and Gunter, 2014 ), and it is chosen as the time point for ERP studies targeting at gesture-speech integration ( Kelly et al, 2004 ; Wu and Coulson, 2005 , 2010 ; Özyürek et al, 2007 ). Similar onsets were also chosen for the investigation of gesture-speech integration for other types of gestures (e.g., emblems and speech) using EEG, fMRI, and simultaneous EEG–fMRI ( He et al, 2015 , 2018 ). The onsets were similarly defined for the two unimodal conditions.…”

“…Departing from previous M/EEG studies that adopting the “mismatch” paradigm (comparing congruent vs. incongruent multimodal stimuli), in which the integration process may be confounded by conflict resolution (c.f., Green et al, 2009 ; Holle et al, 2010 ), we opted to the “additive” paradigm where effects in unimodal conditions (visual-only and auditory-only) were compared to the bimodal condition. This approach of examining multisensory integration has been proven to be highly successful in functional magnetic resonance imaging (fMRI) studies ( Beauchamp et al, 2004 ; Calvert and Thesen, 2004 ; He et al, 2015 , 2018 ), and has recently been adopted to M/EEG research with a focus on brain oscillations ( He et al, 2015 , 2018 ; Drijvers et al, 2018 ). Therefore, we compared the processing of MP co-speech gestures (MPG) with the processing of iconic co-speech gestures (ICG) that semantically integrate concrete (literal) gesture and concrete (literal) speech, which can be semantically less complex.…”

Gesture's body orientation modulates the N400 for visual sentences primed by gestures

“…Literature suggests that human brain is able to effortlessly integrate semantic information from these two input channels. With functional magnetic resonance imaging (fMRI), it is suggested that two regions in the left hemisphere, the left posterior superior temporal sulcus and the left inferior frontal gyrus are crucially involved in the semantic combination and semantic integration between gesture and verbal speech (Dick, Mok, Beharelle, Goldin‐Meadow, & Small, 2012; Green et al, 2009; He et al, 2015, He, Steines, Sommer, et al, 2018; Holle, Obleser, Rueschemeyer, & Gunter, 2010; Kircher et al, 2009; Straube, Green, Bromberger, & Kircher, 2011; Willems, Özyürek, & Hagoort, 2009). With electroencephalography (EEG) methods such as event‐related potentials (ERPs), it has been suggested that the human brain is able to rapidly integrate semantic representations from gesture and language, as reflected by the N400 component.…”

Gesture’s body orientation modulates the N400 during semantic integration of gesture and visual sentence

“…Literature suggests that human brain is able to effortlessly integrate semantic information from these two input channels. With functional magnetic resonance imaging (fMRI), it is suggested that two regions in the left hemisphere, the left posterior superior temporal sulcus and the left inferior frontal gyrus are crucially involved in the semantic combination and semantic integration between gesture and verbal speech (Dick et al, 2012;Green et al, 2009;He et al, 2015;He et al, 2018b;Holle et al, 4 actions (reaching, grasping, and etc.) observed from egocentric vs. allocentric view-point elicited greater sensorimotor-related alpha or beta band power decrease (Angelini et al, 2018;Drew et al, 2015).…”