The interaction of vision and audition in two-dimensional space

Abstract

Using a mouse-driven visual pointer, 10 participants made repeated open-loop egocentric localizations of memorized visual, auditory, and combined visual-auditory targets projected randomly across the two-dimensional frontal field (2D). The results are reported in terms of variable error, constant error and local distortion. The results confirmed that auditory and visual maps of the egocentric space differ in their precision (variable error) and accuracy (constant error), both from one another and as a function of eccentricity and direction within a given modality. These differences were used, in turn, to make predictions about the precision and accuracy within which spatially and temporally congruent bimodal visual-auditory targets are localized. Overall, the improvement in precision for bimodal relative to the best unimodal target revealed the presence of optimal integration well-predicted by the Maximum Likelihood Estimation (MLE) model. Conversely, the hypothesis that accuracy in localizing the bimodal visual-auditory targets would represent a compromise between auditory and visual performance in favor of the most precise modality was rejected. Instead, the bimodal accuracy was found to be equivalent to or to exceed that of the best unimodal condition. Finally, we described how the different types of errors could be used to identify properties of the internal representations and coordinate transformations within the central nervous system (CNS). The results provide some insight into the structure of the underlying sensorimotor processes employed by the brain and confirm the usefulness of capitalizing on naturally occurring differences between vision and audition to better understand their interaction and their contribution to multimodal perception.

Multimodal Integration of Emotional Signals from Voice, Body, and Context: Effects of (In)Congruence on Emotion Recognition and Attitudes Towards Robots

“…MI refers to the processes by which information arriving from one sensory modality interacts with, and sometimes biases, the perception of cues presented in another modality, including how these sensory inputs are combined to yield a unified percept [31][32][33]. MI effects have been studied using behavioural experiments, functional neuroimaging (fMRI) and electroencephalography (EEG) measurements with faces and voices [34][35][36], faces and bodies [37,38], body expression and voices [38,39], and body and sound stimuli [40].…”

Audio Spatial Representation Around the Body

“…Evidence also suggests that space representation is split up into different portions in relation to the body position, i.e., near and far space ( Ladavas and Serino, 2008 ), frontal and rear space ( Saj and Vuilleumier, 2007 ; Viaud-Delmon et al, 2007 ; Zampini et al, 2007 ; Occelli et al, 2011 ), space around specific parts of the body ( di Pellegrino and Làdavas, 2015 ; Serino et al, 2015 ), and space above and below the head in the frontal field ( Finocchietti et al, 2015 ). Studies on neglect patients ( De Renzi et al, 1989 ; Brozzoli et al, 2006 ; Jacobs et al, 2012 ) and on healthy people ( Godfroy-Cooper et al, 2015 ) show that spatial representation can be affected by a specific sensory modality (i.e., vision) and, at the same time, can be intact for other sensory channels (i.e., touch and hearing). Interestingly, studies on agnosia ( Coslett, 2011 ) have shown that object representation could also be selectively impaired with one sense and yet be intact with the others.…”

Sounds are remapped across saccades

“…Nevertheless, inference based on the comparison of saccade trajectory curvature effects across modalities is somewhat limited as we kept stimulus intensity constant, confounding arousal and sensory modality factors. Future work systematically varying the level of the two modalities would better test multi-sensory integration (Godfroy-Cooper, Sandor, Miller, & Welch, 2015).…”