Auditory–Visual Multisensory Interactions in Humans: Timing, Topography, Directionality, and Sources

Abstract

Current models of brain organization include multisensory interactions at early processing stages and within low-level, including primary, cortices. Embracing this model with regard to auditory-visual (AV) interactions in humans remains problematic. Controversy surrounds the application of an additive model to the analysis of event-related potentials (ERPs), and conventional ERP analysis methods have yielded discordant latencies of effects and permitted limited neurophysiologic interpretability. While hemodynamic imaging and transcranial magnetic stimulation studies provide general support for the above model, the precise timing, superadditive/subadditive directionality, topographic stability, and sources remain unresolved. We recorded ERPs in humans to attended, but task-irrelevant stimuli that did not require an overt motor response, thereby circumventing paradigmatic caveats. We applied novel ERP signal analysis methods to provide details concerning the likely bases of AV interactions. First, nonlinear interactions occur at 60 -95 ms after stimulus and are the consequence of topographic, rather than pure strength, modulations in the ERP. AV stimuli engage distinct configurations of intracranial generators, rather than simply modulating the amplitude of unisensory responses. Second, source estimations (and statistical analyses thereof) identified primary visual, primary auditory, and posterior superior temporal regions as mediating these effects. Finally, scalar values of current densities in all of these regions exhibited functionally coupled, subadditive nonlinear effects, a pattern increasingly consistent with the mounting evidence in nonhuman primates. In these ways, we demonstrate how neurophysiologic bases of multisensory interactions can be noninvasively identified in humans, allowing for a synthesis across imaging methods on the one hand and species on the other.

Looming Signals Reveal Synergistic Principles of Multisensory Integration

“…Structurally, monosynaptic projections identified between unisensory (including primary) cortices raise the possibility of interactions during early stimulus processing stages (Falchier et al, 2002(Falchier et al, , 2010Rockland and Ojima, 2003;Cappe and Barone, 2005;Cappe et al, 2009a; see also Beer et al, 2011). In agreement, functional data support the occurrence of multisensory interactions within 100 ms poststimulus onset and within low-level cortical areas (Giard and Peronnet, 1999;Molholm et al, 2002;Martuzzi et al, 2007;Romei et al, 2007Romei et al, , 2009Cappe et al, 2010;Raij et al, 2010;Van der Burg et al, 2011). Nevertheless, the organizing principles governing such multisensory interactions in human cortex and their links to behavior/perception remain largely unresolved.…”

The COGs (context, object, and goals) in multisensory processing

“…The idea that at least the presence of eMSImight be impervious toone's current goals is corroborated by the early (50-100ms) latency of this processand by low-level sensory-perceptual cortices as its likely source (Cappe et al 2010;Raij et al 2000Raij et al , 2010De Meo et al 2015;.Existingresults suggest a surprisingly extensive early cross-talk between inputs from different senses, where auditory-based responses within visual cortices co-occur with or even precede visually-based responses to the same multisensory stimulus (animal models: Schroeder et al 2004;Musacchia and Schroeder 2009;humans: Raij et al 2010;Brang et al 2015). Thus, information is transferred across different senses at latencies still considered as characterising the initial stimulus-driven brain activity, which is thought to be largely independent of top-down control (see e.g.…”

“…Perceptual tasks involving arbitrarily linked multisensory stimuli seem to engage a somewhat different set of brain areas: the STC (likely the synchrony-STC subregion; Stevenson et al 2010) and primary visual and auditory cortices (Martuzzi et al 2007). The STC and low level cortices have also been reported to be functionally coupled (Cappe et al 2010;Werner and Noppeney 2010b).…”

Visual dominance and multisensory integration changes with age

“…Stein et al 2010). Previous event-related potential (ERP) studies in young adults showed that sensory-specific regions, namely primary auditory, primary visual, ventral occipito-temporal and superior temporal cortices displayed increased activity in response to temporally-and semantically-congruent cross-modal inputs (Cappe, Thut, Romei & Murray, 2010;Raij et al, 2010;Teder-Sälejärvi, Russo, McDonald & Hillyard, 2005). Furthermore, intracranial electrophysiological studies (Molholm et al, 2006;Moran, Molholm, Reilly, & Foxe, 2008), magnetoencephalography (MEG) studies (Diaconescu, Alain, & McIntosh, 2011;Raij et al, 2010) and functional neuroimaging studies in humans (Baumann & Greenlee, 2007;Bishop & Miller, 2008;Calvert, Hansen, Iversen, & Brammer, 2001;Grefkes, Weiss, Zilles, & Fink, 2002;Macaluso, George, Dolan, Spence, & Driver, 2004) showed that cross-modal stimuli did not only elicit increased activity in sensory-specific cortices, but also activated a distinct network of posterior parietal brain regions, including the inferior parietal sulcus (IPS), the inferior parietal lobule (IPL), and the superior parietal lobule (SPL).…”