Spatial orienting in complex audiovisual environments

Abstract

Previous studies on crossmodal spatial orienting typically used simple and stereotyped stimuli in the absence of any meaningful context. This study combined computational models, behavioural measures and functional magnetic resonance imaging to investigate audiovisual spatial interactions in naturalistic settings. We created short videos portraying everyday life situations that included a lateralised visual event and a co-occurring sound, either on the same or on the opposite side of space. Subjects viewed the videos with or without eye-movements allowed (overt or covert orienting). For each video, visual and auditory saliency maps were used to index the strength of stimulus-driven signals, and eye-movements were used as a measure of the efficacy of the audiovisual events for spatial orienting. Results showed that visual salience modulated activity in higher-order visual areas, whereas auditory salience modulated activity in the superior temporal cortex. Auditory salience modulated activity also in the posterior parietal cortex, but only when audiovisual stimuli occurred on the same side of space (multisensory spatial congruence). Orienting efficacy affected activity in the visual cortex, within the same regions modulated by visual salience. These patterns of activation were comparable in overt and covert orienting conditions. Our results demonstrate that, during viewing of complex multisensory stimuli, activity in sensory areas reflects both stimulus-driven signals and their efficacy for spatial orienting; and that the posterior parietal cortex combines spatial information about the visual and the auditory modality.

The COGs (context, object, and goals) in multisensory processing

“…Iordanescu et al 2009;van Ee et al 2009;Orchard-Mills et al 2013a, 2013bNardo et al 2014;Mastroberardino et al 2015).While these findings are in line with the influence of feature-based unisensory attention (Desimone and Duncan 1995), whether it applies to multisensory situations remains unclear. To investigate this possibility, Matusz and Eimer (2013) employed multi-stimulus visual displays and instructed participants to search for targets defined by a visual feature alone (e.g.…”

Attention in the real world: toward understanding its neural basis

“…To further approximate the real world, an important avenue for future research will be to study neural mechanisms of visual search in situations that include multiple modalities [102], depth, a realistic temporal structure [61], and active exploration.…”

Not so automatic: Task relevance and perceptual load modulate cross-modal semantic congruence effects on spatial orienting

“…Answering this question can shed some light on the underlying processes supporting cross-modal semantic interactions. Nardo et al (2014) reported that crossmodal semantic congruency between visual events and sounds had no effect on spatial orienting or brain activity (measured with fMRI) when observers watched videos of everyday life scenes. In contrast, another study by Mastroberardino et al (2015), using static images of objects, reported that attention was oriented toward the image semantically congruent (albeit spatially uninformative) sound presented at the same time.…”