A normalization model of multisensory integration

Ohshiro, Tomokazu; Angelaki, Dora E.; DeAngelis, Gregory C.

doi:10.1038/nn.2815

Cited by 281 publications

(283 citation statements)

References 50 publications

Supporting

Mentioning

252

Contrasting

Order By: Relevance

“…More generally, a large body of behavioral literature shows that multisensory perceptual fusion can be robust over extended periods of cross-sensory asynchrony, without any apparent evidence for "cyclic" fluctuations in the coherence of the multisensory percept (5-7). Given the variety of multisensory response types elicited during periods in which stable perceptual fusion should occur, our results underscore the functional role of both enhanced and suppressed spiking responses (43,44). However, this perceptual robustness is in apparent contrast to the observed rhythmicity of neuronal integrative processes (8,9,25,30).…”

Section: Discussionmentioning

confidence: 54%

Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex

Perrodin

Kayser

Logothetis

et al. 2014

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

When social animals communicate, the onset of informative content in one modality varies considerably relative to the other, such as when visual orofacial movements precede a vocalization. These naturally occurring asynchronies do not disrupt intelligibility or perceptual coherence. However, they occur on time scales where they likely affect integrative neuronal activity in ways that have remained unclear, especially for hierarchically downstream regions in which neurons exhibit temporally imprecise but highly selective responses to communication signals. To address this, we exploited naturally occurring face-and voice-onset asynchronies in primate vocalizations. Using these as stimuli we recorded cortical oscillations and neuronal spiking responses from functional MRI (fMRI)-localized voice-sensitive cortex in the anterior temporal lobe of macaques. We show that the onset of the visual face stimulus resets the phase of low-frequency oscillations, and that the face-voice asynchrony affects the prominence of two key types of neuronal multisensory responses: enhancement or suppression. Our findings show a three-way association between temporal delays in audiovisual communication signals, phase-resetting of ongoing oscillations, and the sign of multisensory responses. The results reveal how natural onset asynchronies in cross-sensory inputs regulate network oscillations and neuronal excitability in the voice-sensitive cortex of macaques, a suggested animal model for human voice areas. These findings also advance predictions on the impact of multisensory input on neuronal processes in face areas and other brain regions.H ow the brain parses multisensory input despite the variable and often large differences in the onset of sensory signals across different modalities remains unclear. We can maintain a coherent multisensory percept across a considerable range of spatial and temporal discrepancies (1-4): For example, auditory and visual speech signals can be perceived as belonging to the same multisensory "object" over temporal windows of hundreds of milliseconds (5-7). However, such misalignment can drastically affect neuronal responses in ways that may also differ between brain regions (8-10). We asked how natural asynchronies in the onset of face/voice content in communication signals would affect voice-sensitive cortex, a region in the ventral "object" pathway (11) where neurons (i) are selective for auditory features in communication sounds (12)(13)(14), (ii) are influenced by visual "face" content (12), and (iii) display relatively slow and temporally variable responses in comparison with neurons in primary auditory cortical or subcortical structures (14-16).Neurophysiological studies in human and nonhuman animals have provided considerable insights into the role of cortical oscillations during multisensory conditions and for parsing speech. Cortical oscillations entrain to the slow temporal dynamics of natural sounds (17)(18)(19)(20) and are thought to reflect the excitability of local networks to sensory inputs...

show abstract

Section: Discussionmentioning

confidence: 54%

Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex

Perrodin

Kayser

Logothetis

et al. 2014

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Importantly, this finding is not inconsistent with the hypothesis of altered divisive normalization in autism since cue integration does not theoretically require divisive normalization (42). Altered divisive normalization in autism does, however, predict changes in other forms of multisensory processing that require divisive normalization, such as cue conflict conditions and causal inference (36,38).…”

Section: Multisensory Processingmentioning

confidence: 52%

“…It is also thought to adjust the gain of neural responses to efficiently make use of their dynamic range (15). Divisive normalization further accounts for a number of phenomena in auditory cortex (35), multisensory integration (36), and higher-order processes such as attention (37) and rationality (16). In addition, divisive normalization may be critical for neural circuits to implement marginalization, a type of probabilistic inference that eliminates irrelevant information, so-called "nuisance variables" (38).…”

Section: Connecting the E/i Balance To Neural Computationmentioning

confidence: 99%

A computational perspective on autism

Rosenberg

Patterson

Angelaki

2015

Proc. Natl. Acad. Sci. U.S.A.

147

151

View full text Add to dashboard Cite

Autism is a neurodevelopmental disorder that manifests as a heterogeneous set of social, cognitive, motor, and perceptual symptoms. This system-wide pervasiveness suggests that, rather than narrowly impacting individual systems such as affection or vision, autism may broadly alter neural computation. Here, we propose that alterations in nonlinear, canonical computations occurring throughout the brain may underlie the behavioral characteristics of autism. One such computation, called divisive normalization, balances a neuron's net excitation with inhibition reflecting the overall activity of the neuronal population. Through neural network simulations, we investigate how alterations in divisive normalization may give rise to autism symptomatology. Our findings show that a reduction in the amount of inhibition that occurs through divisive normalization can account for perceptual consequences of autism, consistent with the hypothesis of an increased ratio of neural excitation to inhibition (E/I) in the disorder. These results thus establish a bridge between an E/I imbalance and behavioral data on autism that is currently absent. Interestingly, our findings implicate the context-dependent, neuronal milieu as a key factor in autism symptomatology, with autism reflecting a less "social" neuronal population. Through a broader discussion of perceptual data, we further examine how altered divisive normalization may contribute to a wide array of the disorder's behavioral consequences. These analyses show how a computational framework can provide insights into the neural basis of autism and facilitate the generation of falsifiable hypotheses. A computational perspective on autism may help resolve debates within the field and aid in identifying physiological pathways to target in the treatment of the disorder.A utism is a neurodevelopmental disorder that is dramatically increasing in prevalence (Fig. 1). Recent data place the number of children being diagnosed with autism in the United States at 1 in 68, more than doubling in the last decade (1-4). The disorder is highly pervasive, affecting individuals at cognitive, motor, and perceptual levels. It is furthermore a "spectrum disorder," with symptoms that manifest in varying degrees across individuals. This heterogeneity presents significant challenges to establishing a comprehensive characterization of the disorder.Research investigating the genetic and molecular basis of autism implicates over 100 genes (5), many of which are involved in synaptic development and function (6-8). As such, one prominent hypothesis is that autism arises from a neurophysiological excitation-to-inhibition (E/I) imbalance (9, 10). However, the connection between an E/I imbalance and the behavioral characteristics of the disorder remains unclear. Considering the pervasive nature of autism, and the covariance of loosely related symptoms (11-14), one possibility is that an E/I imbalance widely affects neural computation, in turn giving rise to the broad behavioral symptoms recognized as autism.Here,...

show abstract

“…The topedown generation process of the model drove activity in the sensory units by a weighted sum of memorized/internalized patterns; how much each pattern contributed to the sensory units was determined by the normalized activity of the supramodal recognition units at the top. Such normalization is a candidate basic operation in brain computations (Carandini & Heeger, 2012;Ohshiro, Angelaki, & DeAngelis, 2011).…”

Section: Discussionmentioning

confidence: 99%

Prediction across sensory modalities: A neurocomputational model of the McGurk effect

Olasagasti

Bouton

Giraud

2015

Cortex

View full text Add to dashboard Cite

Audiovisual integrationComputational modeling McGurk effect a b s t r a c tThe McGurk effect is a textbook illustration of the automaticity with which the human brain integrates audio-visual speech. It shows that even incongruent audiovisual (AV) speech stimuli can be combined into percepts that correspond neither to the auditory nor to the visual input, but to a mix of both. Typically, when presented with, e.g., visual /aga/ and acoustic /aba/ we perceive an illusory /ada/. In the inverse situation, however, when acoustic /aga/ is paired with visual /aba/, we perceive a combination of both stimuli, i.e., /abga/ or /agba/. Here we assessed the role of dynamic cross-modal predictions in the outcome of AV speech integration using a computational model that processes continuous audiovisual speech sensory inputs in a predictive coding framework. The model involves three processing levels: sensory units, units that encode the dynamics of stimuli, and multimodal recognition/identity units. The model exhibits a dynamic prediction behavior because evidence about speech tokens can be asynchronous across sensory modality, allowing for updating the activity of the recognition units from one modality while sending topedown predictions to the other modality. We explored the model's response to congruent and incongruent AV stimuli and found that, in the two-dimensional feature space spanned by the speech second formant and lip aperture, fusion stimuli are located in the neighborhood of congruent /ada/, which therefore provides a valid match. Conversely, stimuli that lead to combination percepts do not have a unique valid neighbor. In that case, acoustic and visual cues are both highly salient and generate conflicting predictions in the other modality that cannot be fused, forcing the elaboration of a combinatorial solution.We propose that dynamic predictive mechanisms play a decisive role in the dichotomous perception of incongruent audiovisual inputs.

show abstract

A normalization model of multisensory integration

Cited by 281 publications

References 50 publications

Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex

Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex

A computational perspective on autism

Prediction across sensory modalities: A neurocomputational model of the McGurk effect

Contact Info

Product

Resources

About