Transitions in neural oscillations reflect prediction errors generated in audiovisual speech

Abstract

According to the predictive coding theory, top-down predictions are conveyed by backward connections and prediction errors are propagated forward across the cortical hierarchy. Using MEG in humans, we show that violating multisensory predictions causes a fundamental and qualitative change in both the frequency and spatial distribution of cortical activity. When visual speech input correctly predicted auditory speech signals, a slow delta regime (3-4 Hz) developed in higher-order speech areas. In contrast, when auditory signals invalidated predictions inferred from vision, a low-beta (14-15 Hz) / high-gamma (60-80 Hz) coupling regime appeared locally in a multisensory area (area STS). This frequency shift in oscillatory responses scaled with the degree of audio-visual congruence and was accompanied by increased gamma activity in lower sensory regions. These findings are consistent with the notion that bottom-up prediction errors are communicated in predominantly high (gamma) frequency ranges, whereas top-down predictions are mediated by slower (beta) frequencies.© 2011 Nature America, Inc. All rights reserved.7 9 8 VOLUME 14 | NUMBER 6 | JUNE 2011 nature neurOSCIenCe a r t I C l e S converge-that is, where multisensory predictions are generatedwhereas gamma activity was seen in lower sensory cortices where prediction errors emerge and are propagated forward. RESULTSWe presented 15 subjects with stimuli in one of three conditions: videos (audio-visual: AV condition) of a speaker pronouncing the syllables /pa/, / a/, /la/, /ta/, /ga/ and /fa/ (International Phonetic Alphabet notation); an auditory track of these videos combined with a still face (auditory: A condition); or a mute visual track (visual: V condition). The videos could be either natural or a random combination of auditory and visual tracks, creating conditions in which auditory and visual tracks were congruent (AVc condition) and ones in which they were incongruent (AVi condition; see Online Methods and Supplementary Fig. 1). Incongruent combinations yielding fusion illusory percepts-that is, McGurk stimuli-were excluded 8,11 . Subjects performed an unrelated target detection task on the syllable /fa/ that was presented in A, V or AVc form in 13% of the trials (97% correct detection). These trials were subsequently excluded from the analyses. The five other syllables were chosen because they yielded graded recognition accuracy when presented visually (Fig. 1a), resulting from an increasing predictiveness 10 . The phonological prediction conveyed by mouth movements (visemes) varies in specificity depending on the pronounced syllable. Typically, syllables beginning with a consonant that is formed at the front of the mouth (/p/, /m/) convey a more specific prediction than those formed at the back (/g/, /k/, /r/, /l/) 8 . Our second experimental factor pertained to the validity of the visual prediction with respect to the auditory input. Physically, the audio-visual stimuli could be either congruent (valid prediction) or incongruent (invalid prediction), w...

Prediction across sensory modalities: A neurocomputational model of the McGurk effect

“…Brain activity in early auditory cortex in fusion trials in which participants do give fused responses, is very similar to that observed upon congruent token presentations (Kislyuk, M€ ott€ onen, & Sams, 2008) with differences appearing elsewhere (Erickson et al, 2014). Arnal et al (2011) compared congruent and incongruent non-McGurk AV speech tokens, leading only to combination percepts when incongruent, and found different patterns of brain activation. The visual input showed two kinds of effects on the perceptual process: an early nonspecific effect on auditory cortex and a token-specific effect related to the informative content of the visual modality about the consonant.…”

“…Another difference is our introduction of topedown predictive signals that bias activity at the unimodal level towards patterns compatible with existing representations at the top level. Therefore, our model does not process modalities independently; the processing in one modality influences the processing in the other one, as suggested by electrophysiological data (Arnal, Wyart, & Giraud, 2011;Van Wassenhove et al, 2005.…”

“…Beyond such a straightforward extension, an additional level of processing able to resolve the conflict occurring in combination conditions by explicitly resorting to phonemic-level recognition units could be added. This would allow more precise predictions about the conflict resolution mechanism and would therefore help interpreting the patterns of brain activity in response to incongruent audiovisual speech tokens (Arnal et al, 2011).…”

A Neuronal Model of Predictive Coding Accounting for the Mismatch Negativity

“…As noted by Friston et al (2005), the closely similar neuronal architecture of cortical layers throughout the cerebral cortex supports the view that a similar computational principle of predictive coding may apply to the multiple hierarchical levels of the cortical areas of the brain. Thus, our model may be used to account for higher-order instances of mismatch responses, such as the distinct MMNs evoked by a change in phoneme versus speaker (Giard et al, 1995;Dehaene-Lambertz, 1997), or the mismatch responses observed outside the auditory modality, either in visual (Tales et al, 1999;Pazo-Alvarez et al, 2003), olfactive (Krauel et al, 1999;Pause and Krauel, 2000), and somatosensory (Kekoni et al, 1997;Shinozaki et al, 1998) modalities or even in a crossmodal context (Arnal et al, 2011).…”

Encoding of event timing in the phase of neural oscillations

“…Neural synchronizations in higher frequency ranges (e.g. gamma range, N 40 Hz) provide a reliable index of feature binding within and across sensory modalities (Arnal et al, 2011;Engel et al, 1991;Roelfsema et al, 1997;Senkowski et al, 2008;Tallon-Baudry and Bertrand, 1999). In multisensory integration, low-frequency neural oscillations (delta, 1-2 Hz) play a crucial role in the temporal selection (Besle et al, 2011;Fiebelkorn et al, 2013;Gomez-Ramirez et al, 2011;Lakatos et al, 2008;Schroeder and Lakatos, 2009) and in the integration of AV information (Fiebelkorn et al, 2011;Kösem and van Wassenhove, 2012;Luo et al, 2010).…”