Neural Correlates of Multisensory Integration of Ecologically Valid Audiovisual Events

Abstract

Abstract& A question that has emerged over recent years is whether audiovisual (AV) speech perception is a special case of multisensory perception. Electrophysiological (ERP) studies have found that auditory neural activity (N1 component of the ERP) induced by speech is suppressed and speeded up when a speech sound is accompanied by concordant lip movements. In Experiment 1, we show that this AV interaction is not speechspecific. Ecologically valid nonspeech AV events (actions performed by an actor such as handclapping) were associated with a similar speeding-up and suppression of auditory N1 amplitude as AV speech (syllables). Experiment 2 demonstrated that these AV interactions were not influenced by whether A and V were congruent or incongruent. In Experiment 3 we show that the AV interaction on N1 was absent when there was no anticipatory visual motion, indicating that the AV interaction only occurred when visual anticipatory motion preceded the sound. These results demonstrate that the visually induced speeding-up and suppression of auditory N1 amplitude reflect multisensory integrative mechanisms of AV events that crucially depend on whether vision predicts when the sound occurs. &

Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex

“…However, the extent of the N1 decrease in the right hemisphere varied with the noise level and the behavioral gain, which suggests that the effect was not simply due to unisensory visual activity (since the visual stimuli was the same whatever the noise level). Recently, however, a study reproduced this N1 amplitude decrease using the additive model (Stekelenburg and Vroomen, 2007). Hence, overall, despite methodological problems, the decrease of the auditory N1 component associated with faster processing of audiovisual speech seems to be a robust result.…”

“…Klucharev et al (2003) and Stekelenburg and Vroomen (2007) argued that early interactions in speech perception reflect non-specific integration related to basic computations such as location in space and time because, around these latencies, ERPs are not sensitive to the incongruence of auditory and visual speech cues (Klucharev et al, 2003). At least three arguments undermine this claim: first, it is based on a logical flaw known as converse error.…”

“…1A) and several authors have pointed out that this could partly explain the differences in results between speech and nonspeech studies (e.g. Stekelenburg and Vroomen, 2007).…”

“…However, this last hypothesis is hard to reconcile with the enhancement of behavioral performance generally associated with the N1 suppression effect (although formally, they are not mutually exclusive). Stekelenburg and Vroomen (2007) directly tested the question of speech specificity by comparing the violation of the additive model for speech and non-speech naturalistic stimuli with the same temporal structure as speech (where visual cues preceded auditory cues by about 300 ms). They showed that the decrease in the auditory N1 component was observed in both cases and that it was not observed when using audiovisual stimuli with synchronous onsets.…”

Affective responses to emotional words are boosted in communicative situations

“…de Gelder and Vroomen, 2000) already during early perceptual processing stages (e.g. de Gelder et al, 1999;Gerdes et al, 2013;Pourtois et al, 2000Pourtois et al, , 2002Stekelenburg and Vroomen, 2007) probably involving specialized structures (e.g. de Gelder and Van den Stock, 2011), while incongruent audiovisual input can even lead to perceptual illusions (cf.…”