The Role of Gamma Oscillations During Integration of Metaphoric Gestures and Abstract Speech

Abstract

Metaphoric (MP) co-speech gestures are commonly used during daily communication. They communicate about abstract information by referring to gestures that are clearly concrete (e.g., raising a hand for “the level of the football game is high”). To understand MP co-speech gestures, a multisensory integration at semantic level is necessary between abstract speech and concrete gestures. While semantic gesture-speech integration has been extensively investigated using functional magnetic resonance imaging, evidence from electroencephalography (EEG) is rare. In the current study, we set out an EEG experiment, investigating the processing of MP vs. iconic (IC) co-speech gestures in different contexts, to reveal the oscillatory signature of MP gesture integration. German participants (n = 20) viewed video clips with an actor performing both types of gestures, accompanied by either comprehensible German or incomprehensible Russian (R) speech, or speaking German sentences without any gestures. Time-frequency analysis of the EEG data showed that, when gestures were accompanied by comprehensible German speech, MP gestures elicited decreased gamma band power (50–70 Hz) between 500 and 700 ms in the parietal electrodes when compared to IC gestures, and the source of this effect was localized to the right middle temporal gyrus. This difference is likely to reflect integration processes, as it was reduced in the R language and no-gesture conditions. Our findings provide the first empirical evidence suggesting the functional relationship between gamma band oscillations and higher-level semantic processes in a multisensory setting.

Gesture's body orientation modulates the N400 for visual sentences primed by gestures

“…Twenty‐four participants (15 female, mean age = 24.95, range 19–35) participated in this experiment and were monetarily compensated for participation. The sample size, although appears to be moderate, is comparable to the extensive body of N400 and brain oscillations literature on multimodal language processing (Habets et al, 2011; He, Nagels, Schlesewsky, & Straube, 2018; He, Steines, Sammer, et al, 2018; Wu & Coulson, 2005). All participants were right‐handed as assessed by a questionnaire on handedness (Oldfield, 1971).…”

Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia

“…We performed statistical whole-brain analysis in a two-level, mixed-effects procedure. On the first level, single-participant BOLD responses were modeled by a design matrix comprising the onset time points of each event (critical word of each sentence as used in the previous event-related fMRI and EEG studies, e.g., (He et al, 2015; He et al, 2018a; He et al, 2018c; Straube et al, 2013b; Straube et al, 2018)), with a duration of 5s for all experimental conditions. The micro-time onset was set to the average time bin (8 of 16) to align the onset vector to the slice in the middle of the brain.…”

The Margins of the Language Network in the Brain

“…Both types of gestures may lead to multisensory integration at the semantic level, activating a left frontotemporal network. Furthermore, differential gammaband activity was observed in right temporal cortex for iconic and metaphoric gestures, indicating specific dimensions of higherlevel semantic processing in an audiovisual setting (He et al, 2018). In addition to direct visual involvement in language reception, secondary visual cortex may give rise to visual imagery even in the absence of a visual signal (Bergen et al, 2007), which may also cause cross-modal priming effects (Dils and Boroditsky, 2010).…”