Efficient Visual Search from Synchronized Auditory Signals Requires Transient Audiovisual Events

Abstract

BackgroundA prevailing view is that audiovisual integration requires temporally coincident signals. However, a recent study failed to find any evidence for audiovisual integration in visual search even when using synchronized audiovisual events. An important question is what information is critical to observe audiovisual integration.Methodology/Principal FindingsHere we demonstrate that temporal coincidence (i.e., synchrony) of auditory and visual components can trigger audiovisual interaction in cluttered displays and consequently produce very fast and efficient target identification. In visual search experiments, subjects found a modulating visual target vastly more efficiently when it was paired with a synchronous auditory signal. By manipulating the kind of temporal modulation (sine wave vs. square wave vs. difference wave; harmonic sine-wave synthesis; gradient of onset/offset ramps) we show that abrupt visual events are required for this search efficiency to occur, and that sinusoidal audiovisual modulations do not support efficient search.Conclusions/SignificanceThus, audiovisual temporal alignment will only lead to benefits in visual search if the changes in the component signals are both synchronized and transient. We propose that transient signals are necessary in synchrony-driven binding to avoid spurious interactions with unrelated signals when these occur close together in time.

Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search?

“…Here, it is important to note that the first beep occurred 500-900 ms after display onset, and that participants may have waited for this first beep to start their search (this tendency had already been observed by Van der Burg et al, 2008, and some of our participants admitted to it as well), which means that the effective RTs can be regarded as 750 ms shorter than those reported here.…”

“…Experiment 1 was designed to investigate whether participants' performance in the pip-and-pop task (see Van der Burg et al, 2008) would be influenced by the crossmodal congruency between the pitch of the spatially uninformative auditory cue and the brightness of the associated visual target.…”

“…We decided to use a slightly modified version of the pip-andpop task, a visual search task originally developed by Van der Burg, Olivers, Bronkhorst, and Theeuwes (2008) in which participants' performance is facilitated by the presence of an auditory cue synchronized with color changes of the visual target (see also .…”

“…Lippert, Logothetis, & Kayser, 2007;. On the basis of the combined results of their four experiments, Van der Burg et al (2008) concluded that the temporally synchronous auditory cue was perceptually integrated with the visual target, thus increasing its saliency, and hence making it "pop out" from the distractors (but see , for alternative explanations). Additional evidence for the automaticity of the pip-and-pop effect comes from a recent electroencephalographic study (Van der Burg, Talsma, Olivers, Hickey, & Theeuwes, 2011), in which an early multisensory integration effect was observed (i.e., an audiovisual event-related potential over the left parieto-occipital cortex) occurring within 50-60 ms of stimulus onset that mediated the behavioral search benefits associated with the presence of the auditory cue.…”

Cross-modal associations between vision, touch, and audition influence visual search through top-down attention, not bottom-up capture

“…Auditory and tactile cues have also been shown to influence visual search. This has been shown when the auditory or tactile cue was spatially informative (Bolia, D'Angelo, & McKinley, 1999;Jones, Gray, Spence, & Tan, 2008;Rudmann & Strybel, 1999), when the auditory or tactile cue was temporally synchronous with a change in color of the target (Ngo & Spence, 2010;Van der Burg, Cass, Olivers, Theeuwes, & Alais, 2010;Van der Burg, Olivers, Bronkhorst, & Theeuwes, 2008b, Van der Burg, Olivers, Bronkhorst, & Theeuwes, 2009Zannoli, Cass, Mamassian, & Alais, 2012) and when the auditory cue was semantically congruent with the target object (Iordanescu, Grabowecky, Franconeri, Theeuwes, & Suzuki, 2010;Iordanescu, Gravowecky, & Suzuki, 2011;Iordanescu, Guzman-Martinez, Grabowecky, & Suzuki, 2008).…”