Multisensory Integration: Flexible Use of General Operations

Abstract

Research into the anatomical substrates and “principles” for integrating inputs from separate sensory surfaces has yielded divergent findings. This suggests that multisensory integration is flexible and context-dependent, and underlines the need for dynamically adaptive neuronal integration mechanisms. We propose that flexible multisensory integration can be explained by a combination of canonical, population-level integrative operations, such as oscillatory phase-resetting and divisive normalization. These canonical operations subsume multisensory integration into a fundamental set of principles as to how the brain integrates all sorts of information, and they are being used proactively and adaptively. We illustrate this proposition by unifying recent findings from different research themes such as timing, behavioral goal and experience-related differences in integration.

Behavioral and Neural Foundations of Multisensory Face-Voice Perception in Infancy

“…In adults, neural attenuation for bimodal synchrony is thought to reflect more efficient processing and has been documented in both early sensory and later attentional responses (Belse et al, 2004;Pilling, 2009;van Wassenhove, Grant, & Poepple, 2005), whereas attenuation has only been observed in later going attentional responses in infants (Hyde et al, 2011, Grossmann et al, 2006Reynolds et al, 2014;Vogel et al, 2012). Inconsistencies in the functional brain response between different experimental contexts in infants should not be taken as a limitation of the work, as most current views of multimodal processing and intersensory perception in adults conclude that context has a large effect on how, when, and where the brain is engaged (e.g., van Atteveldt et al, 2014;Matusz, Retsa, & Murray, 2016;Murray, Lewkowicz, Amedi, & Wallace, 2016). However, the fact that neural attenuation for bimodal synchrony has been documented exclusively in later attentional processing in infants but often appears during earlier sensory processing in adults may reflect a genuine developmental difference between infants and adults in efficiency and time course.…”

“…In addition, subcortical regions including the superior colliculus have also been found to be important to multimodal processing in non-human animals and are thought to play similar roles in humans (e.g., Stein, Stanford, Rowland, 2009, 2014. More recently, it has been shown that the particular brain regions involved in intersensory perception in adults depends heavily on the context and type of sensory stimulation (e.g., Talsma, 2015;van Atteveldt, Murray, Thut, & Schroeder, 2014). The questions of which brain regions are involved in multimodal perception in infancy, including multimodal face and voice perception, and whether this changes over development are largely unexplored.…”

“…Research to date has begun to map out when and how multimodal face and voice stimulation is integrated, perceived, and remembered in the adult brain (e.g., Talsma, 2015;van Atteveldt et al, 2014). A comprehensive review of the adult work on multimodal processing and perception is beyond the scope and focus of this article.…”

Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex

“…Cortical oscillations entrain to the slow temporal dynamics of natural sounds (17)(18)(19)(20) and are thought to reflect the excitability of local networks to sensory inputs (21)(22)(23)(24). Moreover, at least in auditory cortex, the onset of sensory input from the nondominant modality can reset the phase of ongoing auditory cortical oscillations (8,25,26), modulating the processing of subsequent acoustic input (8,18,22,(26)(27)(28). Thus, the question arises as to whether and how the phase of cortical oscillations in voicesensitive cortex is affected by visual input.…”

Laminar fMRI: Applications for cognitive neuroscience

“…Influences from one sensory area to the other may arrive in form of feedback projections from multisensory areas (Felleman & Van Essen, 1991) or projected through non-specific thalamic afferents, which are essentially feedforward projections (Van Atteveldt et al, 2014). In addition, there is accumulating evidence that early sensory cortices interact via direct cortico-cortical pathways.…”