Modality-specific attention attenuates visual-tactile integration and recalibration effects by reducing prior expectations of a common source for vision and touch

Abstract

At any moment in time, streams of information reach the brain through the different senses. Given this wealth of noisy information, it is essential that we select information of relevance-a function fulfilled by attention-and infer its causal structure to eventually take advantage of redundancies across the senses. Yet, the role of selective attention during causal inference in cross-modal perception is unknown. We tested experimentally whether the distribution of attention across vision and touch enhances cross-modal spatial integration (visual-tactile ventriloquism effect, Expt. 1) and recalibration (visual-tactile ventriloquism aftereffect, Expt. 2) compared to modalityspecific attention, and then used causal-inference modeling to isolate the mechanisms behind the attentional modulation. In both experiments, we found stronger effects of vision on touch under distributed than under modality-specific attention. Model comparison confirmed that participants used Bayes-optimal causal inference to localize visual and tactile stimuli presented as part of a visual-tactile stimulus pair, whereas simultaneously collected unity judgments-indicating whether the visual-tactile pair was perceived as spatially-aligned-relied on a sub-optimal heuristic. The best-fitting model revealed that attention modulated sensory and cognitive components of causal inference. First, distributed attention led to an increase of sensory noise compared to selective attention toward one modality. Second, attending to both modalities strengthened the stimulusindependent expectation that the two signals belong together, the prior probability of a common source for vision and touch. Yet, only the increase in the expectation of vision and touch sharing a common source was able to explain the observed enhancement of visual-tactile integration and recalibration effects with distributed attention. In contrast, the change in sensory noise explained only a fraction of the observed enhancements, as its consequences vary with the overall level of noise and stimulus congruency. Increased sensory noise leads to enhanced integration effects for visual-tactile pairs with a large spatial discrepancy, but reduced integration effects for stimuli with a small or no cross-modal discrepancy. In sum, our study indicates a weak a priori association

Robust spatial ventriloquism effect and trial-by-trial aftereffect under memory interference

“…Future studies could test alternative manipulations such as more extensive masking stimuli that may provide a more comprehensive sensory-motor interference, or could consider the use of a dual task paradigm enhancing the simultaneous memory load. In fact, two previous studies considered either a dual-task paradigm 32 or diverted attention 25 and found that these did affect, but not abolish, the aftereffect. All in all, more systematic work using experimental manipulations that prove to affect memory in additional control paradigms (known to depend on short-term memory) are required to confirm the observed robustness of the trial-by-trial spatial ventriloquism aftereffect.…”

“…Collectively, the functional analogy of the trial-wise ventriloquism aftereffect with serial dependencies in perceptual decision making and the neuroimaging studies implying medial parietal regions in the aftereffect, make a strong case for a memory-related component in the trial-by-trial ventriloquism aftereffect. However, a number of studies suggested that spatial recalibration may not be easily affected by higher cognitive processes 24 , 25 and in particular studies on the long-term ventriloquism aftereffect have also suggested an independence of memory processes 26 . Overall the literature seems divergent, and most studies focused on the long-term ventriloquism aftereffect.…”

Oculomotor freezing reflects tactile temporal expectation and aids tactile perception

“…Hence, remarkably, our study reveals that oculomotor freezing reflects temporal expectation in the tactile modality. Given that humans do not assume by default that tactile and visual events share a common cause 11 , the tactile modality provides a strong test case that microsaccadic inhibition is a marker of supramodal temporal expectation. Moreover, microsaccades that occurred shortly before, during, or shortly after the target vibration are associated with reduced task performance in tactile discrimination, supporting a functional role of anticipatory microsaccadic inhibition and revealing a crossmodal coupling between the oculomotor system and tactile perception.…”

“…Hence, the oculomotor system might show a preparatory response for auditory events in expectation of an accompanying visual event. In contrast, given that humans do not assume by default that tactile and visual events share a common cause 11 , similar predictions about an upcoming tactile stimulus would not usually trigger visual expectation. Thus, the tactile modality is a strong test case for the possibility of microsaccadic inhibition as a marker of supramodal temporal expectation.…”

Senses of place: architectural design for the multisensory mind

“…Or might it be, perhaps, that in a context in which we are regularly exposed to incongruent environmental/atmospheric multisensory cues - just think of how music is played from loudspeakers without any associated visual referent - that out priors concerning whether to integrate what we see, hear, smell, and feel will necessarily be related, in any meaningful sense, may well be reduced substantially. See Badde, Navarro, and Landy ( 2020 ) and Gau and Noppeney ( 2016 ) on the role of context in the strength of the common-source priors multisensory binding.…”