Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception

Phonetic Recalibration in Audiovisual Speech

“…On this account, interactions in the primary cortex are presumably mediated by the STS via backward projections (Besle et al, 2004). Besides STS, motor regions of planning and execution (Broca's area, premotor cortex, and anterior insula) could be involved via the so-called mirror neurons (e.g., Giard & Peronnet, 1999;Klucharev et al, 2003;Ojanen et al, 2005;Skipper, Nusbaum, & Small, 2005). Broca's area is proposed to be a homologue of the macaque inferior premotor cortex (area F5) where mirror neurons are situated that discharge upon action and perception of goal-directed hand or mouth movements.…”

“…The integration of auditory speech and a lipread-or lexical context has been extensively studied with brain imaging and electrophysiological methods (e.g., Callan et al, 2003;Calvert et al, 1997;Calvert & Campbell, 2003;Campbell, 2008;Colin et al, 2002;Holcomb & Neville, 1990;Klucharev, Möttönen, & Sams, 2003;Sams et al, 1991;van Wassenhove, Grant, & Poeppel, 2005). For instance, lipread speech context modulates auditory speech processing as early as 100 msec after stimulus onset as reflected by the attenuation and speeding-up of the N1 component in the ERPs (Besle, Fort, Delpuech, & Giard, 2004;Klucharev et al, 2003;van Wassenhove et al, 2005) whereas lexically induced modulation of auditory speech processing is often reported to occur at around 400 msec (e.g., Holcomb & Neville, 1990). However, there is accumulative evidence that the early effects of lipread speech reflect low-level visual prediction (i.e., the anticipatory visual motion warns the listener about when a sound is going to occur) rather than higher-level phonetic integration that presumably occurs later in time (e.g., Vroomen & Stekelenburg, 2010).…”

“…Additionally, an auditory /ba/ sound combined with lipread /ga/ will be perceived as a /da/ because the audiovisual (hence AV) perceptual conflict is solved by fusing the inputs from both modalities into one percept (McGurk & MacDonald, 1976). By using Electroencephalography (EEG), it has been demonstrated that lipread speech affects auditory processing in the auditory cortex as early as ~100 -200 msec after sound onset (Besle et al, 2004;Klucharev et al, 2003;van Wassenhove et al, 2005). More specifically, the auditory N1 component in the recorded Event Related Potentials (ERPs) is attenuated (Besle et al, 2004;Klucharev et al, 2003) andspeeded-up (van Wassenhove et al, 2005) by simultaneously delivered lipread input.…”

Brief Report: Can You See What is Not There? Low-level Auditory–visual Integration in Autism Spectrum Disorder

“…ERP studies indicate that this processing usually occurs later in time, and is therefore associated with higher order processing (see e.g. Klucharev, Mottonen, & Sams, 2003;Lebib et al, 2004). However, there is accumulating evidence that multimodal integration also includes the modulation of activity at cortical brain sites that used to be considered modality specific and are usually related to perceptual aspects of processing (Calvert et al, 1997(Calvert et al, , 1999.…”

Interest and validity of the additive model in electrophysiological studies of multisensory interactions

“…The [AV-(A+V)] complex was used to determine cortical regions that were uniquely activated by polysensory stimulation." (Barth et al, 1995 p.179) Although this model theoretically can be applied to any measures of human brain activity, it has been mainly used in electrophysiological data (scalp ERP and magneto-encephalography, MEG: Miniussi et al, 1998;Giard & Peronnet, 1999;Foxe et al, 2000;Raij et al, 2000;Fort et al, 2002aFort et al, , 2002bMolholm et al, 2002;Klucharev et al, 2003;Möttönen et al, 2004). On the other hand, its use has been recurrently criticized (Teder-Sälejärvi et al, 2002;Calvert & Thesen, in press) because of the multiple biases it can generate in the estimation of the crossmodal interactions if several important conditions are not fulfilled.…”