Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

Abstract

In a naturalistic environment, auditory cues are often accompanied by information from other senses, which can be redundant with or complementary to the auditory information. Although the multisensory interactions derived from this combination of information and that shape auditory function are seen across all sensory modalities, our greatest body of knowledge to date centers on how vision influences audition. In this review, we attempt to capture the state of our understanding at this point in time regarding this topic. Following a general introduction, the review is divided into 5 sections. In the first section, we review the psychophysical evidence in humans regarding vision’s influence in audition, making the distinction between vision’s ability to enhance versus alter auditory performance and perception. Three examples are then described that serve to highlight vision’s ability to modulate auditory processes: spatial ventriloquism, cross-modal dynamic capture, and the McGurk effect. The final part of this section discusses models that have been built based on available psychophysical data and that seek to provide greater mechanistic insights into how vision can impact audition. The second section reviews the extant neuroimaging and far-field imaging work on this topic, with a strong emphasis on the roles of feedforward and feedback processes, on imaging insights into the causal nature of audiovisual interactions, and on the limitations of current imaging-based approaches. These limitations point to a greater need for machine-learning-based decoding approaches toward understanding how auditory representations are shaped by vision. The third section reviews the wealth of neuroanatomical and neurophysiological data from animal models that highlights audiovisual interactions at the neuronal and circuit level in both subcortical and cortical structures. It also speaks to the functional significance of audiovisual interactions for two critically important facets of auditory perception—scene analysis and communication. The fourth section presents current evidence for alterations in audiovisual processes in three clinical conditions: autism, schizophrenia, and sensorineural hearing loss. These changes in audiovisual interactions are postulated to have cascading effects on higher-order domains of dysfunction in these conditions. The final section highlights ongoing work seeking to leverage our knowledge of audiovisual interactions to develop better remediation approaches to these sensory-based disorders, founded in concepts of perceptual plasticity in which vision has been shown to have the capacity to facilitate auditory learning.

Integration of somatosensory and motor-related information in the auditory system

“…Although most perceptual studies of multisensory integration have focused on vision and audition ( Opoku-Baah et al, 2021 ), interactions can occur across all sensory modalities and there is growing evidence for close links between auditory and tactile processing. Over the past 20 years, numerous studies have explored the effects of auditory-tactile interactions on human perception.…”

“…Vision and hearing provide the primary source of information about distant objects and events that may be too far away to engage the other senses. Furthermore, visual and auditory cues arising from the same source are often complementary and integrated by the brain in ways that shape our perception of the information provided by the source, including its location, timing and identity (see Opoku-Baah et al, 2021 for a recent review). The visual and auditory systems are equally important for processing inputs from nearby sources, such as when we are reading text on or listening to our mobile phones.…”

The effect of semantic content on the perception of audiovisual movieclips

“…The findings of the current study can be implemented in various settings. Education approaches already apply methods that not only take advantage of the enhancement effect induced by audiovisual information (Opoku-Baah et al, 2021), but also consider the semantic associations that words carry. For instance, when introducing a difficult topic or vocabulary to the students, educators use the Keyword Method in order to help students memorize the new word or content by visualizing it i.e.…”

Lipreading: A Review of Its Continuing Importance for Speech Recognition With an Acquired Hearing Loss and Possibilities for Effective Training

“…It is clear that the perceptual integration of auditory and visual speech stimuli is more than the sum of unisensory auditory and visual speech perception (e.g., Grant & Seitz, 1998 ; Opoku-Baah et al, 2021 ; Ross et al, 2007 ). The mechanisms of AV speech integration and how and why they differ from unisensory AO and VO processing is far beyond the scope of this review article.…”