Multisensory integration processes underlying speech perception as revealed by the McGurk illusion

Marques, Lucas Murrins; Lapenta, Olívia Morgan; Costa, Tássia R.; Boggio, Paulo S.

doi:10.1080/23273798.2016.1190023

Cited by 12 publications

(6 citation statements)

References 104 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proportion of the reported McGurk percepts can be used as a measure of the extent that perceivers tend to use visual information in audiovisual speech perception. That is, the increase of the McGurk proportion indicates that the perceivers tend to make more use of visual information (for review, see Alsius, Paré, & Munhall, 2018;Marques, Lapenta, Costa, & Boggio, 2016).…”

Section: Introductionmentioning

confidence: 99%

A value-driven McGurk effect: Value-associated faces enhance the influence of visual information on audiovisual speech perception and its eye movement pattern

Luo

Kang

Guo

et al. 2020

Atten Percept Psychophys

View full text Add to dashboard Cite

This study investigates whether and how value-associated faces affect audiovisual speech perception and its eye movement pattern. Participants were asked to learn to associate particular faces with or without monetary reward in the training phase, and, in the subsequent test phase, to identify syllables that the talkers had said in video clips in which the talkers' faces had or had not been associated with reward. The syllables were either congruent or incongruent with the talkers' mouth movements. Crucially, in some cases, the incongruent syllables could elicit the McGurk effect. Results showed that the McGurk effect occurred more often for reward-associated faces than for non-reward-associated faces. Moreover, the signal detection analysis revealed that participants had lower criterion and higher discriminability for reward-associated faces than for non-reward-associated faces. Surprisingly, eye movement data showed that participants spent more time looking at and fixated more often on the extraoral (nose/cheek) area for reward-associated faces than for non-reward-associated faces, while the opposite pattern was observed on the oral (mouth) area. The correlation analysis demonstrated that, over participants, the more they looked at the extraoral area in the training phase because of reward, the larger the increase of McGurk proportion (and the less they looked at the oral area) in the test phase. These findings not only demonstrate that value-associated faces enhance the influence of visual information on audiovisual speech perception but also highlight the importance of the extraoral facial area in the value-driven McGurk effect.

show abstract

Section: Introductionmentioning

confidence: 99%

A value-driven McGurk effect: Value-associated faces enhance the influence of visual information on audiovisual speech perception and its eye movement pattern

Luo

Kang

Guo

et al. 2020

Atten Percept Psychophys

View full text Add to dashboard Cite

show abstract

“…More than forty years ago, McGurk and MacDonald (1976) published a remarkable (and now famous) example of visual influence on auditory speech perception: when an auditory stimulus (e.g., /ba/) was presented with the face of a talker articulating a different syllable (e.g., /ga/), listeners often experienced an illusory percept distinct from both sources (e.g., /da/). Since that time, McGurk stimuli have been used in countless studies of audiovisual integration in humans (not to mention the multitude of classroom demonstrations on multisensory processing) (Marques, Lapenta, Costa, & Boggio, 2016). At the same time, the stimuli typically used to elicit a McGurk effect differ substantially from what we typically encounter in conversation.…”

mentioning

confidence: 99%

Audiovisual speech perception: Moving beyond McGurk

Engen¹,

Dey²,

Sommers³

et al. 2019

Preprint

View full text Add to dashboard Cite

Although listeners use both auditory and visual cues during speech perception, the cognitive and neural bases for their integration remain a matter of debate. One common approach to measuring multisensory integration is to use McGurk tasks, in which discrepant auditory and visual cues produce auditory percepts that differ from those based solely on unimodal input. Not all listeners show the same degree of susceptibility to the McGurk illusion, and these individual differences in susceptibility are frequently used as a measure of audiovisual integration ability. However, despite their popularity, we argue that McGurk tasks are ill-suited for studying the kind of multisensory speech perception that occurs in real life: McGurk stimuli are often based on isolated syllables (which are rare in conversations) and necessarily rely on audiovisual incongruence that does not occur naturally. Furthermore, recent data show that susceptibility on McGurk tasks does not correlate with performance during natural audiovisual speech perception. Although the McGurk effect is a fascinating illusion, truly understanding the combined use of auditory and visual information during speech perception requires tasks that more closely resemble everyday communication.

show abstract

“…One important aspect to consider is that speech–gesture integration is an automatic process of extraction and combination of AV information to generate the percept of what the interlocutor intends to communicate (Soto-Faraco et al 2004b ). This process is so strong that mismatching facial and auditory information generates audiovisual illusions, such as the McGurk effect (McGurk and MacDonald 1976 , see Marques et al 2016 for a review). It is also noteworthy that in Drijvers et al ( 2021 ) the auditory stimuli were modulated in amplitude, whereas we modulated the pitch, since sound amplitude modulation was used to generate the panning for the perception of sound movement .…”

Section: Discussionmentioning

confidence: 99%

Spatial and temporal (non)binding of audiovisual rhythms in sensorimotor synchronisation

et al. 2023

Self Cite

View full text Add to dashboard Cite

Human movement synchronisation with moving objects strongly relies on visual input. However, auditory information also plays an important role, since real environments are intrinsically multimodal. We used electroencephalography (EEG) frequency tagging to investigate the selective neural processing and integration of visual and auditory information during motor tracking and tested the effects of spatial and temporal congruency between audiovisual modalities. EEG was recorded while participants tracked with their index finger a red flickering (rate fV = 15 Hz) dot oscillating horizontally on a screen. The simultaneous auditory stimulus was modulated in pitch (rate fA = 32 Hz) and lateralised between left and right audio channels to induce perception of a periodic displacement of the sound source. Audiovisual congruency was manipulated in terms of space in Experiment 1 (no motion, same direction or opposite direction), and timing in Experiment 2 (no delay, medium delay or large delay). For both experiments, significant EEG responses were elicited at fV and fA tagging frequencies. It was also hypothesised that intermodulation products corresponding to the nonlinear integration of visual and auditory stimuli at frequencies fV ± fA would be elicited, due to audiovisual integration, especially in Congruent conditions. However, these components were not observed. Moreover, synchronisation and EEG results were not influenced by congruency manipulations, which invites further exploration of the conditions which may modulate audiovisual processing and the motor tracking of moving objects.

show abstract

Multisensory integration processes underlying speech perception as revealed by the McGurk illusion

Cited by 12 publications

References 104 publications

A value-driven McGurk effect: Value-associated faces enhance the influence of visual information on audiovisual speech perception and its eye movement pattern

A value-driven McGurk effect: Value-associated faces enhance the influence of visual information on audiovisual speech perception and its eye movement pattern

Audiovisual speech perception: Moving beyond McGurk

Spatial and temporal (non)binding of audiovisual rhythms in sensorimotor synchronisation

Contact Info

Product

Resources

About