Intersensory binding across space and time: A tutorial review

Abstract

Spatial ventriloquism refers to the phenomenon that a visual stimulus such as a flash can attract the perceived location of a spatially discordant but temporally synchronous sound. An analogous example of mutual attraction between audition and vision has been found in the temporal domain, where temporal aspects of a visual event, such as its onset, frequency, or duration, can be biased by a slightly asynchronous sound. In this review, we examine various manifestations of spatial and temporal attraction between the senses (both direct effects and aftereffects), and we discuss important constraints on the occurrence of these effects. Factors that potentially modulate ventriloquism-such as attention, synesthetic correspondence, and other cognitive factors-are described. We trace theories and models of spatial and temporal ventriloquism, from the traditional unity assumption and modality appropriateness hypothesis to more recent Bayesian and neural network approaches. Finally, we summarize recent evidence probing the underlying neural mechanisms of spatial and temporal ventriloquism.

Rapid temporal recalibration occurs crossmodally without stimulus specificity but is absent unimodally

“…Two different prevailing theories are, first, that the process occurs via sensory-specific temporal shifts and, second, that realignment is a function of the adaptation of dedicated multisensory timing neurons (Di Luca et al, 2009;Navarra et al, 2009;Roach et al, 2011;Chen and Vroomen, 2013). According to the sensory-specific temporal shift account, exposure to crossmodal temporal asynchrony results in temporal recalibration via adjustments to processing speeds of the component sensory modalities relative to one another to realign them towards synchrony (Di Luca et al, 2009;Navarra et al, 2009).…”

“…Temporal recalibration also helps in a similar way, as following long adaptation phases to asynchronous crossmodal stimuli the perceived onset asynchrony is reduced (Fujisaki et al, 2004;Hanson et al, 2008;Keetels and Vroomen, 2008). In other words, the point of subjective simultaneity (PSS) shifts towards the adapted asynchrony, perhaps due to changes in the processing speeds within single modalities (Di Luca et al, 2009;Navarra et al, 2009), or perhaps due to adaptation of dedicated multisensory timing neurons that code for various temporal asynchronies (Roach et al, 2011;Chen and Vroomen, 2013).…”

“…The challenge when using synchrony as a cue for multisensory combination is how to preserve temporal relationships between events in the external world in the neural signals they elicit (Vroomen and Keetels, 2010;Chen and Vroomen, 2013). For instance, travel times for light and sound energy differ markedly and once received by the perceiver, processing latencies differ between sensory modalities (Alais and Carlile, 2005;Vroomen and Keetels, 2010;Arnold and Yarrow, 2011).…”

Anticipating action effects recruits audiovisual movement representations in the ventral premotor cortex

“…Because the integration of motor and sensory aspects of movements during simulation seems to be essential for anticipation (for reviews, see Shmuelof & Zohary, 2005;Zentgraf et al, 2011), one can also ask how far simulation involves representations in more than one sensory modality, that is, multisensory representations. When events have to be perceived with high temporal precision, critical information can be gained from the auditory sensory modality in interaction with vision (see, for a review, Chen & Vroomen, 2013;Murray & Spierer, 2009). Hence, anticipating a goal-directed action in order to prepare an appropriate reaction may be susceptible to cross-modal effects.…”

Differential sensory cortical involvement in auditory and visual sensorimotor temporal recalibration: Evidence from transcranial direct current stimulation (tDCS)

“…This suggests a critical role of the auditory cortex in timing across different stimulus modalities (Guttman, Gilroy, & Blake, 2005;Grahn, Henry, & McAuley, 2011;Kanai, Lloyd, Bueti, & Walsh, 2011;Meyer, Baumann, Marchina, & Jancke, 2007;Sugano et al, 2012). Therefore, the auditory cortex might be a candidate for amodal time regulation (analogous to visual cortex role in auditory spatial perception, Lewald, Meister, Weidemann, & Töpper, 2004;Zimmer, Lewald, Erb, Grodd, & Karnath, 2004; also see ventriloquism effect, Chen & Vroomen, 2013) and its role might extend to temporal recalibration process. However, this hypothesis has not been tested directly using brain stimulation.…”