Prediction across sensory modalities: A neurocomputational model of the McGurk effect

Abstract

Audiovisual integrationComputational modeling McGurk effect a b s t r a c tThe McGurk effect is a textbook illustration of the automaticity with which the human brain integrates audio-visual speech. It shows that even incongruent audiovisual (AV) speech stimuli can be combined into percepts that correspond neither to the auditory nor to the visual input, but to a mix of both. Typically, when presented with, e.g., visual /aga/ and acoustic /aba/ we perceive an illusory /ada/. In the inverse situation, however, when acoustic /aga/ is paired with visual /aba/, we perceive a combination of both stimuli, i.e., /abga/ or /agba/. Here we assessed the role of dynamic cross-modal predictions in the outcome of AV speech integration using a computational model that processes continuous audiovisual speech sensory inputs in a predictive coding framework. The model involves three processing levels: sensory units, units that encode the dynamics of stimuli, and multimodal recognition/identity units. The model exhibits a dynamic prediction behavior because evidence about speech tokens can be asynchronous across sensory modality, allowing for updating the activity of the recognition units from one modality while sending topedown predictions to the other modality. We explored the model's response to congruent and incongruent AV stimuli and found that, in the two-dimensional feature space spanned by the speech second formant and lip aperture, fusion stimuli are located in the neighborhood of congruent /ada/, which therefore provides a valid match. Conversely, stimuli that lead to combination percepts do not have a unique valid neighbor. In that case, acoustic and visual cues are both highly salient and generate conflicting predictions in the other modality that cannot be fused, forcing the elaboration of a combinatorial solution.We propose that dynamic predictive mechanisms play a decisive role in the dichotomous perception of incongruent audiovisual inputs.

Audio-visual combination of syllables involves time-sensitive dynamics following from fusion failure

“…While the mechanisms leading to AV speech fusion are relatively well understood, those leading to AV stimulus combination are still unknown. Based on a previous computational model, we conjectured that AV combination follows from the difficulty to map the auditory and visual physical features in a multisensory space presumably located in the LSTS 28 . AV combination would hence result in a more demanding processing sequence than AV fusion, Figure 1B).…”

“…However, according to our predictive model of AV syllable integration 28 an interesting illustration of how predictive coding could apply to AV integration 28,57,58 .…”

Integrating prediction errors at two time scales permits rapid recalibration of speech sound categories

“…As described above it proposes that syllables are encoded in terms of the expected amplitudes and variances of audiovisual features. The expected amplitudes were taken from the mean values across 10 productions from a single male speaker (Olasagasti, Bouton, and Giraud 2015), the amplitudes were then normalized by dividing by the highest value for each feature. The remaining parameters in the model, variances and sensory noise levels, were chosen so that the overall categorization results, percentage of /aba/, /ada/ and /aga/ responses to the 6 types of stimuli were qualitatively similar to those reported by Lüttke and colleagues.…”

“…The lip and 2 nd formant temporal modulation profiles (M V (t) and M A (t), Fig 1B) were defined as in (Olasagasti, Bouton, and Giraud 2015). Each profile, representing the intervocalic transition between the two vowels in the "a" vocalic context, was modelled with 289 time points.…”

“…To test our hypothesis we simulated Lüttke et al's (Lüttke et al 2016) experiment, using a model of audiovisual integration based on hierarchical Bayesian inference. We explicitly included a model for the McGurk effect based on our previous work (Olasagasti, Bouton, and Giraud 2015) and supplemented it with an adaptation mechanism that uses residual prediction error to update the internal representation -in the form of a hierarchical generative model-associated with the perceived category. The model divides the process in two steps; the first one is an on-line (moment-by-moment) prediction error minimization during the perceptual inference process, at the end of which there might remain "residual" prediction errors.…”