Beat gestures modulate auditory integration in speech perception

Biau, Emmanuel; Soto‐Faraco, Salvador

doi:10.1163/187847612x647054

Cited by 24 publications

(47 citation statements)

References 10 publications

(14 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…That is, perhaps participants accurately encoded the speech in all conditions, but when explicitly forced to decide on the directionality of the intonation they heard, only then did gesture enter the picture and affect the decision-making process. Future studies using a more direct measure of online processing, such as taking RTs or ERPs (Biau & Soto-Faraco, 2013) during the actual encoding phase, would address this subtle, but important, distinction.…”

Section: Obligatory Integrationmentioning

confidence: 99%

Metaphoric Gestures Facilitate Perception of Intonation More than Length in Auditory Judgments of Non-Native Phonemic Contrasts

Kelly

Bailey

Hirata

2017

Collabra: Psychology

View full text Add to dashboard Cite

It is well established that hand gestures affect comprehension and learning of semantic aspects of a foreign language (FL). However, much less is known about the role of hand gestures in lower-level language processes, such as perception of phonemes. To address this gap, we explored the role that metaphoric gestures play in perceiving FL speech sounds that varied on two dimensions: length and intonation. English speaking adults listened to Japanese length contrasts and sentence-final intonational distinctions in the context of congruent, incongruent and no gestures. For intonational contrasts, identification was more accurate for congruent gestures and less accurate for incongruent gestures relative to the baseline no gesture condition. However, for the length contrasts, there was no such clear and consistent pattern, and in fact, congruent gestures made speech processing more effortful. We conclude that metaphoric gestures help with some-but not all-novel speech sounds in a FL, suggesting that gesture and speech are phonemically integrated to differing extents depending on the nature of the gesture and/or speech sound.

show abstract

Section: Obligatory Integrationmentioning

confidence: 99%

Metaphoric Gestures Facilitate Perception of Intonation More than Length in Auditory Judgments of Non-Native Phonemic Contrasts

Kelly

Bailey

Hirata

2017

Collabra: Psychology

View full text Add to dashboard Cite

show abstract

“…Recent ERP evidence suggests that beat gestures facilitate speech processing at various linguistic levels and serve as attention cues during the comprehension of natural speech (Biau & Soto-Faraco, 2013). In the ERP signal, words accompanied by beat gestures give rise to two positive effects, which have been attributed to early sensory processing (before 100 msec after target onset), as well as to the phonological analysis of the target word (P2 component around 200 msec after target onset).…”

Section: Beat Gestures Modulate Online Speech Processing and Serve Asmentioning

confidence: 99%

“…In the ERP signal, words accompanied by beat gestures give rise to two positive effects, which have been attributed to early sensory processing (before 100 msec after target onset), as well as to the phonological analysis of the target word (P2 component around 200 msec after target onset). Biau and Soto-Faraco (2013) suggested that beat gestures serve as highlighters, which modulate the attentional state of the listener and guide their attention to relevant information in the speech signal. However, the study focused on the role of beat gestures without examining the acoustic aspects of words accompanied by beat gestures.…”

Section: Beat Gestures Modulate Online Speech Processing and Serve Asmentioning

confidence: 99%

“…Previous ERP studies do not provide unambiguous evidence that a particular ERP effect is specific for beat gesture but not for other types of hand movement. For example, Biau and Soto-Faraco (2013) did not use a control gesture, and the attention effect of beat gesture they reported could be the pure result of visual emphasis caused by moving the hand, rather than the result of a specific function of beat gesture. Holle et al (2012) added a moving dot on the screen as a control condition to beat gesture.…”

Section: Selecting An Appropriate Control Hand Movement Condition To mentioning

confidence: 99%

“…This led to the prediction that, in trials without a gesture, focused elements would capture attention and elicit a P300-like positivity (Dimitrova et al, 2012;Cowles et al, 2007;Bornkessel et al, 2003). Because beat gestures also capture attention (Biau & Soto-Faraco, 2013;Wang & Chu, 2013;Holle et al, 2012), we hypothesized that they would modulate the early sensory stages of visual processing and give rise to an N1 and/or a P2/P300 component. Third, we hypothesized that the processing of beat gestures may interact with information structure.…”

Section: The Present Studymentioning

confidence: 99%

See 2 more Smart Citations

Beat that Word: How Listeners Integrate Beat Gesture and Focus in Multimodal Speech Discourse

Dimitrova

Chu

Lin³

et al. 2016

Journal of Cognitive Neuroscience

View full text Add to dashboard Cite

Abstract■ Communication is facilitated when listeners allocate their attention to important information (focus) in the message, a process called "information structure." Linguistic cues like the preceding context and pitch accent help listeners to identify focused information. In multimodal communication, relevant information can be emphasized by nonverbal cues like beat gestures, which represent rhythmic nonmeaningful hand movements. Recent studies have found that linguistic and nonverbal attention cues are integrated independently in single sentences. However, it is possible that these two cues interact when information is embedded in context, because context allows listeners to predict what information is important. In an ERP study, we tested this hypothesis and asked listeners to view videos capturing a dialogue. In the critical sentence, focused and nonfocused words were accompanied by beat gestures, grooming hand movements, or no gestures. ERP results showed that focused words are processed more attentively than nonfocused words as reflected in an N1 and P300 component. Hand movements also captured attention and elicited a P300 component. Importantly, beat gesture and focus interacted in a late time window of 600-900 msec relative to target word onset, giving rise to a late positivity when nonfocused words were accompanied by beat gestures. Our results show that listeners integrate beat gesture with the focus of the message and that integration costs arise when beat gesture falls on nonfocused information. This suggests that beat gestures fulfill a unique focusing function in multimodal discourse processing and that they have to be integrated with the information structure of the message. ■

show abstract

Gesture's body orientation modulates the N400 for visual sentences primed by gestures

Luell

Muralikrishnan

et al. 2020

Human Brain Mapping

View full text Add to dashboard Cite

Body orientation of gesture entails social‐communicative intention, and may thus influence how gestures are perceived and comprehended together with auditory speech during face‐to‐face communication. To date, despite the emergence of neuroscientific literature on the role of body orientation on hand action perception, limited studies have directly investigated the role of body orientation in the interaction between gesture and language. To address this research question, we carried out an electroencephalography (EEG) experiment presenting to participants (n = 21) videos of frontal and lateral communicative hand gestures of 5 s (e.g., raising a hand), followed by visually presented sentences that are either congruent or incongruent with the gesture (e.g., “the mountain is high/low…”). Participants underwent a semantic probe task, judging whether a target word is related or unrelated to the gesture‐sentence event. EEG results suggest that, during the perception phase of handgestures, while both frontal and lateral gestures elicited a power decrease in both the alpha (8–12 Hz) and the beta (16–24 Hz) bands, lateral versus frontal gestures elicited reduced power decrease in the beta band, source‐located to the medial prefrontal cortex. For sentence comprehension, at the critical word whose meaning is congruent/incongruent with the gesture prime, frontal gestures elicited an N400 effect for gesture‐sentence incongruency. More importantly, this incongruency effect was significantly reduced for lateral gestures. These findings suggest that body orientation plays an important role in gesture perception, and that its inferred social‐communicative intention may influence gesture‐language interaction at semantic level.

show abstract

Beat gestures modulate auditory integration in speech perception

Cited by 24 publications

References 10 publications

Metaphoric Gestures Facilitate Perception of Intonation More than Length in Auditory Judgments of Non-Native Phonemic Contrasts

Metaphoric Gestures Facilitate Perception of Intonation More than Length in Auditory Judgments of Non-Native Phonemic Contrasts

Beat that Word: How Listeners Integrate Beat Gesture and Focus in Multimodal Speech Discourse

Gesture's body orientation modulates the N400 for visual sentences primed by gestures

Contact Info

Product

Resources

About