Multisensory Processing in the Auditory Cortex

Abstract

The capacity of the brain to combine and integrate information provided by the different sensory systems has a profound impact on perception and behavior. This is especially the case for audition, with many studies demonstrating that the ability of listeners to detect, discriminate, or localize sounds can be altered in the presence of other sensory cues. For example, the availability of congruent visual stimuli can make it easier to localize sounds or to understand speech, benefits that are most apparent when auditory signals are weak or degraded by the presence of background noise. Multisensory convergence has been demonstrated at most levels of the auditory pathway, from the cochlear nucleus to the auditory cortex. This is particularly the case in extralemniscal nuclei from the midbrain upward but has also been observed in the tonotopically organized lemniscal or core projections. In addition to inheriting multisensory signals from subcortical levels, the auditory cortex receives visual and somatosensory inputs from other cortical areas. Although nonauditory stimuli can evoke spiking activity in auditory cortex, they typically modulate auditory responses. These interactions appear to provide contextual cues that signal the presence of an upcoming sound, but they can also increase the information conveyed by cortical neurons about the location or identity of sounds and may even recalibrate cortical responses when the information provided by different sensory modalities is conflicting. Identifying the neural circuitry responsible for the behavioral consequences of multisensory integration remains an area of intense investigation.

Visual modulation of firing and spectrotemporal receptive fields in mouse auditory cortex

“…Environmental events are often transduced by multiple sensory modalities, subserving multisensory perceptual processes such as spatial localization and communication. Although multisensory integration was once believed to be mediated primarily by higher-order cortices, mounting evidence indicates that earlier stations, including primary sensory cortices, extensively integrate information from other sensory and motor areas ( Wallace et al, 2004 ; Ghazanfar and Schroeder, 2006 ; Budinger and Scheich, 2009 ; Smiley and Falchier, 2009 ; Kajikawa et al, 2012 ; Stehberg et al, 2014 ; Schneider and Mooney, 2018 ; King et al, 2019 ). In primary auditory cortex (A1), visual influences have been widely observed in many species and preparations, likely reflecting the pervasiveness of audiovisual interaction in natural behavior ( Wallace et al, 2004 ; Banks et al, 2011 ; Budinger and Scheich, 2009 ; Stehberg et al, 2014 ; King et al, 2019 ).…”

“…Although multisensory integration was once believed to be mediated primarily by higher-order cortices, mounting evidence indicates that earlier stations, including primary sensory cortices, extensively integrate information from other sensory and motor areas ( Wallace et al, 2004 ; Ghazanfar and Schroeder, 2006 ; Budinger and Scheich, 2009 ; Smiley and Falchier, 2009 ; Kajikawa et al, 2012 ; Stehberg et al, 2014 ; Schneider and Mooney, 2018 ; King et al, 2019 ). In primary auditory cortex (A1), visual influences have been widely observed in many species and preparations, likely reflecting the pervasiveness of audiovisual interaction in natural behavior ( Wallace et al, 2004 ; Banks et al, 2011 ; Budinger and Scheich, 2009 ; Stehberg et al, 2014 ; King et al, 2019 ). For instance, anatomical studies report direct projections from visual areas ( Banks et al, 2011 ; Budinger and Scheich, 2009 ; Stehberg et al, 2014 ) and physiological studies have observed neurons with visual responses or visually modulated responses to sound ( Bizley et al, 2007 ; Bizley and King, 2008 ; Kayser et al, 2009 , 2010 ; Kobayasi et al, 2013 ).…”

Visual modulation of spectrotemporal receptive fields in mouse auditory cortex

“…Evidence accumulated within recent decades demonstrates that primary auditory cortex (A1) is not exclusively involved in processing sound. Instead, A1 integrates information carried by projections from multiple sensory and motor areas with input from the ascending auditory pathway (Schneider and Mooney, 2018; King et al, 2019). For many species including humans, monkeys, and mice, visual projections comprise a particularly dense source of input to A1 (Banks et al, 2011).…”

Visuospatial attention revamps cortical processing of sound: restrict stimulus uncertainty

“…This may require that the temporal structure of visual input is effective at modulating neural activity at the auditory cortex for instance. Evidence suggests that limiting the temporal coherence of AV objects may constrain how visual spatial attention influences auditory processing (Beauchamp, 2019;King et al, 2019;Van der Burg et al, 2008). Yet, it is currently unknown how are auditory populations hierarchically and functionally organized to interact with visual signals and effect such changes.…”

“…A possibility is that, under AV attention, multimodal object formation may be subsequent to visuospatial selection (Talsma & Woldorff, 2005). There is evidence that by limiting cross-modal temporal coherence one reduces the effectiveness of visuospatial attention to influence auditory processing (Beauchamp, 2019; King et al, 2019; Van der Burg et al, 2008). In addition, full temporal coherence with a dynamic visual stimulus has been shown to promote auditory segregation behaviorally (Maddox et al, 2015).…”