Visual Anticipatory Information Modulates Multisensory Interactions of Artificial Audiovisual Stimuli

Abstract

The neural activity of speech sound processing (the N1 component of the auditory ERP) can be suppressed if a speech sound is accompanied by concordant lip movements. Here we demonstrate that this audiovisual interaction is neither speech specific nor linked to humanlike actions but can be observed with artificial stimuli if their timing is made predictable. In Experiment 1, a pure tone synchronized with a deformation of a rectangle induced a smaller auditory N1 than auditory-only presentations if the temporal occurrence of this audiovisual event was made predictable by two moving disks that touched the rectangle. Local autoregressive average source estimation indicated that this audiovisual interaction may be related to integrative processing in auditory areas. When the moving disks did not precede the audiovisual stimulus--making the onset unpredictable--there was no N1 reduction. In Experiment 2, the predictability of the leading visual signal was manipulated by introducing a temporal asynchrony between the audiovisual event and the collision of moving disks. Audiovisual events occurred either at the moment, before (too "early"), or after (too "late") the disks collided on the rectangle. When asynchronies varied from trial to trial--rendering the moving disks unreliable temporal predictors of the audiovisual event--the N1 reduction was abolished. These results demonstrate that the N1 suppression is induced by visual information that both precedes and reliably predicts audiovisual onset, without a necessary link to human action-related neural mechanisms.

Phonetic Recalibration in Audiovisual Speech

“…Given that in AV speech, lipread input usually precedes the auditory signal (e.g., Chandrasekaran, Trubanova, Stillittano, Caplier, & Ghazanfar, 2009), Stekelenburg and Vroomen (2007) argued that the visually-induced N1 modulations arise whenever the visual input precedes the audio, thus warning the listener about when the sound is going to occur. This was corroborated by similar results obtained with artificial AV stimuli in which anticipatory visual motion reliably predicted sound onset (Vroomen & Stekelenburg, 2010). It thus appears that the temporal characteristics of an AV stimulus in which the visual component precedes the audio are responsible for the early N1 effects, irrespective of whether the stimuli are ecologically valid or artificial.…”

“…In line with earlier reports, (e.g., Besle et al, 2004;Fort, Delpuech, Pernier, & Giard, 2002;Giard & Peronnet, 1999;Klucharev et al, 2003;Vroomen & Stekelenburg, 2010), this allowed us to compare the audiovisual (AV -V) with the auditory-only (A) condition and interpret any difference as an integration effect between the two modalities. In a first analysis, we compared both groups on the N1 and P2 peaks to reveal lipread-induced modulations that reflect visual prediction (N1) and possibly, a phonetic mechanism (P2).…”

“…For instance, lipread speech context modulates auditory speech processing as early as 100 msec after stimulus onset as reflected by the attenuation and speeding-up of the N1 component in the ERPs (Besle, Fort, Delpuech, & Giard, 2004;Klucharev et al, 2003;van Wassenhove et al, 2005) whereas lexically induced modulation of auditory speech processing is often reported to occur at around 400 msec (e.g., Holcomb & Neville, 1990). However, there is accumulative evidence that the early effects of lipread speech reflect low-level visual prediction (i.e., the anticipatory visual motion warns the listener about when a sound is going to occur) rather than higher-level phonetic integration that presumably occurs later in time (e.g., Vroomen & Stekelenburg, 2010). The experiment in…”

“…Some have reported that visual speech may affect auditory processing as early as the auditory cortex (Calvert et al, 1997;Colin et al, 2002;Möttönen, Krause, Tiippana, & Sams, 2002;Pekkola et al, 2005;Sams et al, 1991). The interaction has been found to occur between 150 and 250 ms using the mismatch negativity paradigm (Colin et al, 2002;Möttönen et al, 2002;Sams et al, 1991), while others have reported that as early as 100 ms, the auditory N1 component is attenuated and speeded up when auditory speech is accompanied by lipread information (Besle et al, 2004;van Wassenhove et al, 2005), possibly because visual speech predicts when a sound is going to occur Vroomen & Stekelenburg, 2010). Notably though, to date it is not known whether auditory speech also affects visual processing of lipread speech.…”

“…Previous work has demonstrated that, in addition to the N1 modulations, lipread speech elicits an attenuation of the auditory P2 . Although the auditory P2 reduction by visual input can also be elicited with artificial stimuli, it is argued to be functionally dissociated from N1 (Vroomen & Stekelenburg, 2010).…”