Event related spectral perturbations of gesture congruity: Visuospatial resources are recruited for multimodal discourse comprehension

Momsen, Jacob; Gordon, Jared; Wu, Ying Choon; Coulson, Seana

doi:10.1016/j.bandl.2021.104916

Cited by 6 publications

(7 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We found that participants’ visuospatial WM capacities were predictive of their reliance on the gestural channel. This finding is in line with those of previous research, indicating the role of visuospatial WM in processing gestures (e.g., Özer & Göksun, 2020a; Momsen et al, 2021; Wu & Coulson, 2014a; Wu et al, 2022). As gestures are spatial representations (Alibali, 2005; Arslan & Göksun, 2021), understanding them requires spatial processing, which inevitably recruits visuospatial resources.…”

Section: Discussionsupporting

confidence: 93%

“…Finally, we observed great variability in channel choice in participants in Experiment 1. This finding is well aligned with prior studies showing individual differences in gesture processing (Aldugom et al, 2020; Momsen et al, 2021; Özer & Göksun, 2020a; Wu & Coulson, 2014a, 2014b; Wu et al, 2022). While Experiment 1 showcased this variability in participants’ learning from feedback, Experiment 2 demonstrated that the variability was not entirely learning-related; it was also present when listeners were left free to choose their preferred channel for information processing.…”

Section: Discussionsupporting

confidence: 91%

“…Moreover, Wu and Coulson (2014a) showed that individual differences in visuospatial WM, but not in verbal WM, predicted the benefit from adding related gestures to speech (see also Wu et al, 2022). The authors suggested that the visuospatial WM memory might be responsible for processing gestures and information integration from that channel with speech (see also Momsen et al, 2021).…”

Section: Processing Information From Two Channelsmentioning

confidence: 99%

See 2 more Smart Citations

Trust my gesture or my word: How do listeners choose the information channel during communication?

Arslan¹,

Ng²,

Göksun³

et al. 2024

Journal of Experimental Psychology: Learning, Memory, and Cogni

View full text Add to dashboard Cite

Information can be conveyed via multiple channels such as verbal and gestural (visual) channels during communication. Sometimes the information from different channels does not match (e.g., saying right while pointing to the left). How do addressees choose which information to act upon in such cases? In two experiments, we investigated this issue by having participants follow instructions on how to move objects on the screen. Experiment 1 examined whether people’s choice of channel can be altered by feedback favoring either the verbal or the gestural channel. In Experiment 2, there was no feedback and participants were free to choose either channel. We also assessed participants’ verbal and visuospatial working memory capacities. Results showed that, when faced with contradicting information, there is a natural bias at the group level toward relying on the verbal channel, although this bias can be temporarily altered by probabilistic feedback. Moreover, when labels were shorter and of higher frequency, participants relied more on the verbal channel. In the absence of feedback, the capacity of individuals’ visual, but not verbal, working memory determined reliance on one channel versus the other. Collectively, these results show that information selection in communication is influenced by group-level biases, as well as the properties of items and characteristics of individuals.

show abstract

Section: Discussionsupporting

confidence: 93%

Section: Discussionsupporting

confidence: 91%

Section: Processing Information From Two Channelsmentioning

confidence: 99%

See 1 more Smart Citation

Trust my gesture or my word: How do listeners choose the information channel during communication?

Arslan¹,

Ng²,

Göksun³

et al. 2024

Journal of Experimental Psychology: Learning, Memory, and Cogni

View full text Add to dashboard Cite

show abstract

“…This is in agreement with a previous study showing alpha and beta power suppression in the precentral gyrus regions during motor imagery ( De Lange et al, 2008 ). An alpha and beta suppression in frontal regions ( Momsen et al, 2021 ) and more specifically in the left IFG and left pSTS/MTG, STG regions ( Drijvers et al, 2018 ) is consistent with their role in gesture-speech integration highlighted by imaging studies. An increase in gamma power in the left temporal lobe was found at the presentation of the gesture’s stroke and co-occurring speech, suggesting an attempt to integrate both information ( Drijvers et al, 2018 ).…”

Section: Investigating the Relationship Between Iconic Gestures And Languagesupporting

confidence: 78%

The Role of Iconic Gestures in Speech Comprehension: An Overview of Various Methodologies

et al. 2021

View full text Add to dashboard Cite

Iconic gesture-speech integration is a relatively recent field of investigation with numerous researchers studying its various aspects. The results obtained are just as diverse. The definition of iconic gestures is often overlooked in the interpretations of results. Furthermore, while most behavioral studies have demonstrated an advantage of bimodal presentation, brain activity studies show a diversity of results regarding the brain regions involved in the processing of this integration. Clinical studies also yield mixed results, some suggesting parallel processing channels, others a unique and integrated channel. This review aims to draw attention to the methodological variations in research on iconic gesture-speech integration and how they impact conclusions regarding the underlying phenomena. It will also attempt to draw together the findings from other relevant research and suggest potential areas for further investigation in order to better understand processes at play during speech integration process.

show abstract

“…Belgrimet S delved into the complexities of multimodal discourse analysis on posters for advocacy campaigns against violence against women and tried to explore how to construct posters with different symbols when cooperating with language [9]. Momsen J observed videos of speakers describing objects to demonstrate the role of visuospatial working memory in understanding multimodal discourse and proved the role of multimodality in discourse [10]. Based on the concept of foothold, Chen J activated the footsteps of different judges through multi-mode resources, including animators, animators + principals, and determined the footsteps of judges in the court [11].…”

Section: Introductionmentioning

confidence: 99%

Investigation of Environmental Multimodal Discourse Evaluation in English Reading Teaching

2024

curtm

View full text Add to dashboard Cite

English reading ability is one of the important components of a student's English ability, which is an important criterion to measure the student's English level. The cultivation of reading ability plays an important role in promoting the improvement of English proficiency. The emergence of environmental multimodal discourse analysis teaching method has brought about important changes in English reading teaching, which reflects the application in English reading teaching. This paper compared the class using environmental multimodal discourse analysis teaching method with the class using traditional teaching mode. It was concluded that the application of environmental multimodal discourse analysis teaching method to English reading teaching improved students' interest in English reading in college. The average was about 2 points higher than that of traditional teaching and the average English reading score was also increased by 11 points, which satisfied students' individual reading needs and improved students' English reading scores.

show abstract

Event related spectral perturbations of gesture congruity: Visuospatial resources are recruited for multimodal discourse comprehension

Cited by 6 publications

References 79 publications

Trust my gesture or my word: How do listeners choose the information channel during communication?

Trust my gesture or my word: How do listeners choose the information channel during communication?

The Role of Iconic Gestures in Speech Comprehension: An Overview of Various Methodologies

Investigation of Environmental Multimodal Discourse Evaluation in English Reading Teaching

Contact Info

Product

Resources

About