Audiovisual short-term influences and aftereffects in motion: Examination across three sets of directional pairings

Abstract

The study of cross-modal influences in perception, particularly between the auditory and visual modalities, has been intensified recently. This paper reports on a comprehensive study of auditory-visual cross-modal influences in motion, including motion aftereffects (MAE). We examined both auditory influences on visual perception and vice versa. Visual motion interactions were examined using three directional pairings or configurations: along the horizontal, vertical, and depth axes. In Experiment 1 we assessed how the simultaneous presence of a strong motion signal in one modality affected the perception of motion in the other modality. To investigate further whether such influences have long-term effects, we tested whether adaptation in one modality alone could produce cross-modal MAEs in Experiment 2. Overall, the pattern of results was similar across all directional pairings, with the strongest cross-modal influences observed in motion along the horizontal axis; this is likely due to the greater co-localization of the two stimuli in this configuration. Although both auditory and visual stimuli affected the other modality when presented simultaneously, significant cross-modally induced aftereffects could only be produced using visual stimuli. However, we did observe vertical visual MAE following adaptation to auditory spectral motion. These results are discussed in terms of current psychophysical and neurophysiological findings concerning the way in which auditory-visual signals are processed.

The effects of stereo disparity on the behavioural and electrophysiological correlates of perception of audio–visual motion in depth

“…Together, the results of Experiment 1 are in line with studies showing visual modulation of audio motion in depth perception (Harrison, 2012;Jain et al, 2008). Similarly, enhanced behavioural responses for looming visual stimuli have previously been reported in unimodal (Ball & Tronick, 1971;Schiff et al, 1962) and multimodal (Cappé et al, 2009;Harrison, 2012) paradigms, and have been interpreted to be a result of the adaptive salience of approaching objects.…”

Music can elicit a visual motion aftereffect

“…Using RDK stimuli marks an important deviation from the previous literature on cross-modal MAEs (e.g., Jain et al, 2008), as RDK stimuli lack any trackable visual features and have a much richer spatiotemporal frequency content. Each RDK stimulus consisted of 100 moving dots.…”

“…It is thus possible that the pitch-verticality correspondence is a stronger metaphoric mapping, and thus can elicit a visual MAE from purely auditory adaptation. This is one of the hypotheses tested by Jain, Sally, and Papathomas (2008). They found that after adaptation to continuously rising or falling auditory motion, participants were more likely to judge ambiguous sinusoidal gratings as moving in the opposite direction.…”

Computational model for interactions between auditory and visual motion mechanisms

“…Neurophysiological and imaging studies have enabled scientist to clarify the role of multisensory areas in the brain, like the SC [2]. We have developed a computational model that explains the interactions between auditory and visual motion mechanisms as observed by Jain et al [1]. These interactions are mediated by the feedback connections from the integrative multimodal areas (SC) to the unimodal areas in the brain.…”

“…

Jain et al [1] showed that a strong motion signal in one modality (visual/auditory) influences the perception of a simultaneously present weak motion signal in the other modality (auditory/visual) for motion along the three cardinal axes. Furthermore, they also observed auditory crossmodal motion aftereffects (MAE) for all directions and visual crossmodal MAE for vertical direction of motion.

…”