The effects of stereo disparity on the behavioural and electrophysiological correlates of perception of audio–visual motion in depth

Abstract

Motion is represented by low-level signals, such as size-expansion in vision or loudness changes in the auditory modality. The visual and auditory signals from the same object or event may be integrated and facilitate detection. We explored behavioural and electrophysiological correlates of congruent and incongruent audio-visual depth motion in conditions where auditory level changes, visual expansion, and visual disparity cues were manipulated. In Experiment 1 participants discriminated auditory motion direction whilst viewing looming or receding, 2D or 3D, visual stimuli. Responses were faster and more accurate for congruent than for incongruent audio-visual cues, and the congruency effect (i.e., difference between incongruent and congruent conditions) was larger for visual 3D cues compared to 2D cues. In Experiment 2, event-related potentials (ERPs) were collected during presentation of the 2D and 3D, looming and receding, audio-visual stimuli, while participants Page 2 of 44 detected an infrequent deviant sound. Our main finding was that audio-visual congruity was affected by retinal disparity at an early processing stage (135 -160 ms) over occipito-parietal scalp. Topographic analyses suggested that similar brain networks were activated for the 2D and 3D congruity effects, but that cortical responses were stronger in the 3D condition.Differences between congruent and incongruent conditions were observed between 140 -200 ms, 220 -280 ms, and 350 -500 ms after stimulus onset.

Temporal Audiovisual Motion Prediction in 2D- vs. 3D-Environments

“…Importantly, we found a significant interaction for visual congruence with stimulation dimension in experiment 1 (detection): RTs were shorter for visually congruent stimulation in pseudo-3D sessions than in real-3D sessions. This is in obvious contrast to some previous research: e.g., González et al (2010) suggested that disparity is an effective cue for motion perception in depth and Harrison et al (2015) even observed that facilitation effects appear to be stronger in real-3D environments than in experimental set-ups without 3D stimulation. However, in comparison to Harrison et al (2015) we did not use a discrimination task for auditory motion direction and did not focus on accuracies but rather a detection task with RTs.…”

“…This is in obvious contrast to some previous research: e.g., González et al (2010) suggested that disparity is an effective cue for motion perception in depth and Harrison et al (2015) even observed that facilitation effects appear to be stronger in real-3D environments than in experimental set-ups without 3D stimulation. However, in comparison to Harrison et al (2015) we did not use a discrimination task for auditory motion direction and did not focus on accuracies but rather a detection task with RTs. Possibly, a more pronounced congruence effect in a 3D environment depends on a task which requires the in-depth analysis of spatial properties and is only reflected in answer quality but not in reaction speed during temporal detection.…”

“…Due to the relevance of binocular cues to motion perception in depth ( Cumming and Parker, 1994 ; Brooks and Stone, 2004 ; Rokers et al, 2009 ) three-dimensional (3D) stimulation, induced by visual disparity could be another critical ecological factor governing these differences ( Kitagawa and Ichihara, 2002 ; Zannoli et al, 2012 ; Ogawa and Macaluso, 2013 ; Ogawa et al, 2013 ; Gaebler et al, 2014 ; Harrison et al, 2015 ). In fMRI studies distinct activation patterns were found while watching audiovisual movements ( Ogawa and Macaluso, 2013 ) or movies ( Ogawa et al, 2013 ) in 3D compared to 2D condition.…”

“…Adaption to a visual looming stimulus led to a motion after effect for a stationary sound ( Kitagawa and Ichihara, 2002 ), participants responded faster to bimodal than unimodal looming stimuli ( Cappe et al, 2009 ) and there is also an advantage in visual search in depth if a search is accompanied by a congruent sound ( Zannoli et al, 2012 ). Furthermore, the congruency effect (higher accuracies for audiovisual congruent looming conditions) has been reported to be more pronounced during stereoscopic 3D than 2D stimulation ( Harrison et al, 2015 ).…”

Aging Impairs Audiovisual Facilitation of Object Motion Within Self-Motion

“…The stimulus used in the experiment contained no stereo disparity and used only two monocular cues to depth. A recent study reported that a sound moving in depth had a greater impact on detection of visual motion in depth when the visual stimulus contained stereo disparity (Harrison et al, 2015). Similarly, congruent vestibular cues had a more robust effect on perception of heading from optic flow when visual stimuli contained stereo disparity, compared to only 2D information (Butler et al, 2011).…”