An auditory-visual tradeoff in susceptibility to clutter

Abstract

Sensory cortical mechanisms combine auditory or visual features into perceived objects. This is difficult in noisy or cluttered environments. Knowing that individuals vary greatly in their susceptibility to clutter, we wondered whether there might be a relation between an individual’s auditory and visual susceptibilities to clutter. In auditory masking, background sound makes spoken words unrecognizable. When masking arises due to interference at central auditory processing stages, beyond the cochlea, it is called informational masking. A strikingly similar phenomenon in vision, called visual crowding, occurs when nearby clutter makes a target object unrecognizable, despite being resolved at the retina. We here compare susceptibilities to auditory informational masking and visual crowding in the same participants. Surprisingly, across participants, we find a negative correlation (R = –0.7) between susceptibility to informational masking and crowding: Participants who have low susceptibility to auditory clutter tend to have high susceptibility to visual clutter, and vice versa. This reveals a tradeoff in the brain between auditory and visual processing.

What the Neuroscience and Psychology of Magic Reveal about Misinformation

“…For example, items or patterns can blend perceptually with adjacent ones and become unrecognizable, a phenomenon known as “visual crowding” [ 42 , 43 ]. Similarly, sounds can become unrecognizable when preceded and followed by nonoverlapping noise, a phenomenon known as “auditory masking” [ 41 , 44 ].…”

Individual differences in speech-on-speech masking are correlated with cognitive and visual task performance

“…Finding directly comparable tasks in the two modalities also is challenging due to differences in how the stimuli are perceptually segregated and how attention is engaged and applied in the specific tasks. The visual tasks used by Clayton et al (2016) and by Zhang et al (2021) were in fact quite different.…”

“…An important part of the design in both the auditory and visual tasks employed by Clayton et al was that the conditions were intentionally high in uncertainty and, therefore, presumably high in informational masking (IM) in the auditory domain and, by analogy, high in “IM” in the visual domain. The term “informational masking” commonly is applied to auditory task limitations (e.g., Kidd et al , 2008b ) but potentially could be generalized to visual tasks that are high in uncertainty ( Zhang et al , 2021 ).…”

“… Zhang et al (2021) compared performance on auditory tasks high in IM [in contrast to energetic masking (EM); see Culling and Stone (2017) for a review] with both speech and non-speech stimuli to performance measured in a visual crowding paradigm considered to be a visual analog of auditory IM (i.e., determining how distinguishable a target object was in the peripheral field of vision from other adjacent distracting objects; Pelli, 2008 ). The Zhang et al (2021) study found that performance across individuals was significantly negatively correlated for these auditory and visual tasks, indicating that, paradoxically, the subjects who were better in overcoming IM on the auditory task were worse on the high-IM visual task. This unexpected result led the authors to conclude—in contrast to the interpretation of the findings of Clayton et al noted above—that an individual's “tolerance for clutter” extends only to one domain.…”

“…This conclusion suggests that the effects of IM are in fact modality-specific, which may be considered paradoxical if it is thought that both tasks reflect the limitations of the same central mechanism (e.g., selective attention) that is influenced by uncertainty. It is also possible, however, that the auditory and visual IM tasks used by Zhang et al (2021) , while comparable in many ways, did not engage the same central mechanism(s). For example, speech intelligibility tasks tap linguistic processing to a degree that depends on factors, such as syntax and semantic content, that may be subject to influences not readily apparent in nonlinguistic visual tasks.…”