The facilitative effect of gestures on the neural processing of semantic complexity in a continuous narrative

Cuevas, Paulina; Steines, Miriam; He, Yifei; Nagels, Arne; Culham, Jody C.; Straube, Benjamin

doi:10.1016/j.neuroimage.2019.03.054

Cited by 20 publications

(20 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite reduced neural processing of both social and non-social content in gesture and speech modalities, patients displayed intact neural processing of these features, as well as improved task accuracy in the bimodal conditions. This enhancement effect concurs with a line of proposals (Holler & Levinson, 2019), who argue for a bi-directional facilitative relation between speech and gesture (for empirical evidence, see (Cuevas et al, 2019; Drijvers et al, 2018; He et al, 2018b; Krahmer & Swerts, 2007; Wang & Chu, 2013)). More importantly, our finding extends previous basic research, suggesting the translational implication of this mechanism.…”

Section: Discussionsupporting

confidence: 90%

“…For the processing of concrete and non-social information, a previous study shows that patients are impaired in visually presented sentences (Kuperberg et al, 2008). Moreover, given potential dysfunctional processing of social or non-social information in schizophrenia, it remains unclear if these deficits could be potentially compensated by multimodal inputs containing both speech and gesture: Emerging literature suggests mutual facilitation between speech and gesture, at least for healthy populations (Cuevas et al, 2019; Drijvers et al, 2018; He et al, 2018b; Krahmer & Swerts, 2007; Wang & Chu, 2013).…”

Section: Introductionmentioning

confidence: 99%

“…We focused directly on group differences between a group of patients suffering from schizophrenia or schizoaffective disorder, and their age- and education-matched controls: for social content, as patients are well-known for their social cognition impairments, we expected patients to show reduced activation in the mPFC, irrespective of encoding modality (Frith, 2004); For non-social content, despite mixed findings from previous neuroimaging research on hand action observation on schizophrenia (Horan et al, 2014; Thakkar et al, 2014), following previous report on dysfunctional processing of non-social linguistic stimuli in schizophrenia (Kuperberg et al, 2008), we hypothesized neural modulation of the object-related regions for patients with schizophrenia for both gesture and speech modalities. Additionally, based on prior basic research on mutual facilitation between gesture and speech (Cuevas et al, 2019; see Holler & Levinson, 2019 for review), we hypothesized that the bimodal input could compensate for potential unimodal processing deficits, leading to improved performance in the patient group.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia

Steines

Sammer

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Background: Schizophrenia (SZ) is characterized by marked social dysfunctions encompassing potential deficits in the processing of social and non-social information, especially in everyday settings where multiple modalities are present. To date, the neurobiological basis of these deficits remains elusive.Methods: In a functional magnetic resonance imaging (fMRI) study, 17 patients with schizophrenia or schizoaffective disorder, and 18 matched controls watched videos of an actor speaking, gesturing (unimodal), and both speaking and gesturing (bimodal) about social or non-social events in a naturalistic way. Participants had to judge whether each video contains person-related (social) or object-related (non-social) information.Results: When processing social content, controls activated the medial prefrontal cortex (mPFC) for both speech and gesture conditions; patients, in comparison to controls, showed no different activation in the speech condition but reduced activation in the mPFC in the gesture condition. For non-social content, across modalities, controls recruited the bilateral pre/postcentral gyrus, superior temporal gyrus, and insula, as well as the left occipitotemporal cortex; patients showed reduced activation of the left postcentral gyrus and the right insula only in the speech condition. Moreover, in the bimodal conditions, patients displayed improved task performance and comparable activation to controls in both social and non-social content. Conclusions:Patients with SZ displayed modality-specific aberrant neural processing of social and non-social information, which is not present for the bimodal conditions. This finding provides novel insights into dysfunctional social cognition in SZ, and may have potential therapeutic implications.

show abstract

Section: Discussionsupporting

confidence: 90%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia

Steines

Sammer

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…It is fairly established that gesture and language both share comparable neural underpinnings (Xu, Gannon, Emmorey, Smith, & Braun, 2009), and that the two communicative channels interact with each other (Holler & Levinson, 2019). Neural evidence from both M/EEG and fMRI also shows that gesture modulates language processing, most commonly in a facilitative manner (Biau & Soto‐Faraco, 2013; Cuevas et al, 2019; Drijvers et al, 2018b; Zhang, Frassinelli, Tuomainen, Skipper, & Vigliocco, 2020). Therefore, given the social nature of communicative gestures, it might be hypothesized that gestures differing in social aspects, as in the case of facing versus not facing the addressee, may differentially affect the semantic processing of gesture as well as the semantic prediction/integration between gesture and language.…”

Section: Introductionmentioning

confidence: 99%

Gesture's body orientation modulates the N400 for visual sentences primed by gestures

Luell

Muralikrishnan

et al. 2020

Human Brain Mapping

Self Cite

View full text Add to dashboard Cite

Body orientation of gesture entails social‐communicative intention, and may thus influence how gestures are perceived and comprehended together with auditory speech during face‐to‐face communication. To date, despite the emergence of neuroscientific literature on the role of body orientation on hand action perception, limited studies have directly investigated the role of body orientation in the interaction between gesture and language. To address this research question, we carried out an electroencephalography (EEG) experiment presenting to participants (n = 21) videos of frontal and lateral communicative hand gestures of 5 s (e.g., raising a hand), followed by visually presented sentences that are either congruent or incongruent with the gesture (e.g., “the mountain is high/low…”). Participants underwent a semantic probe task, judging whether a target word is related or unrelated to the gesture‐sentence event. EEG results suggest that, during the perception phase of handgestures, while both frontal and lateral gestures elicited a power decrease in both the alpha (8–12 Hz) and the beta (16–24 Hz) bands, lateral versus frontal gestures elicited reduced power decrease in the beta band, source‐located to the medial prefrontal cortex. For sentence comprehension, at the critical word whose meaning is congruent/incongruent with the gesture prime, frontal gestures elicited an N400 effect for gesture‐sentence incongruency. More importantly, this incongruency effect was significantly reduced for lateral gestures. These findings suggest that body orientation plays an important role in gesture perception, and that its inferred social‐communicative intention may influence gesture‐language interaction at semantic level.

show abstract

“…It is fairly established that gesture and language both share comparable neural underpinnings (Xu et al, 2009), and that the two communicative channels interact with each other (Holler & Levinson, 2019). Neural evidence from both M/EEG and fMRI also shows that gesture modulates language processing, most commonly in a facilitative manner (Biau & Soto-Faraco, 2013;Cuevas et al, 2019;Drijvers et al, 2018b;Zhang et al, 2020). Therefore, given the social nature of communicative gestures, it might be hypothesized that gestures differing in social aspects, as in the case of facing vs. not facing the addressee, may differentially affect the semantic processing of gesture as well as the semantic integration between gesture and language.…”

Section: Introductionmentioning

confidence: 99%

Gesture’s body orientation modulates the N400 during semantic integration of gesture and visual sentence

Luell

Muralikrishnan

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Body orientation of gesture entails social-communicative intention, and may thus influence how gestures are perceived and comprehended together with auditory speech during face-to-face communication. To date, despite the emergence of neuroscientific literature on the role of body orientation on hand action perception, limited studies have directly investigated the role of body orientation in the interaction between gesture and language. To address this research question, we carried out an EEG experiment presenting to participants (n=21) videos of frontal and lateral hand gestures of five-seconds (e.g., raising a hand), followed by visually presented sentences that are either congruent or incongruent with the hand gesture (e.g., 'the mountain is high/low…'). All participants underwent a semantic-probe task, judging whether a target word is related or unrelated to the gesture-speech event. EEG results suggest that, during the perception phase of handgestures, while both frontal and lateral gestures elicited power decrease in both the alpha (8-12Hz) and the beta (16-24Hz) bands, lateral gestures elicited reduced power decrease in the beta band when compared with frontal gestures. For sentence comprehension, at the critical word whose meaning is congruent/incongruent with the gesture, frontal gestures elicited an N400 effect for gesture-sentence incongruency. More importantly, this incongruency effect was significantly reduced for lateral gestures. The findings suggest that body orientation plays a crucial role in gesture perception, and that its inferred socialcommunicative intention influences gesture-sentence semantic integration in an interactive manner.

show abstract

The facilitative effect of gestures on the neural processing of semantic complexity in a continuous narrative

Cited by 20 publications

References 79 publications

Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia

Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia

Gesture's body orientation modulates the N400 for visual sentences primed by gestures

Gesture’s body orientation modulates the N400 during semantic integration of gesture and visual sentence

Contact Info

Product

Resources

About