Sounds enhance visual completion processes

Abstract

Everyday vision includes the detection of stimuli, figure-ground segregation, as well as object localization and recognition. Such processes must often surmount impoverished or noisy conditions; borders are perceived despite occlusion or absent contrast gradients. These illusory contours (ICs) are an example of so-called mid-level vision, with an event-related potential (ERP) correlate at ∼100-150 ms post-stimulus onset and originating within lateral-occipital cortices (the IC). Presently, visual completion processes supporting IC perception are considered exclusively visual; any influence from other sensory modalities is currently unknown. It is now well-established that multisensory processes can influence both low-level vision (e.g. detection) as well as higher-level object recognition. By contrast, it is unknown if mid-level vision exhibits multisensory benefits and, if so, through what mechanisms. We hypothesized that sounds would impact the IC. We recorded 128-channel ERPs from 17 healthy, sighted participants who viewed ICs or no-contour (NC) counterparts either in the presence or absence of task-irrelevant sounds. The IC was enhanced by sounds and resulted in the recruitment of a distinct configuration of active brain areas over the 70-170 ms post-stimulus period. IC-related source-level activity within the lateral occipital cortex (LOC), inferior parietal lobe (IPL), as well as primary visual cortex (V1) were enhanced by sounds. Moreover, the activity in these regions was correlated when sounds were present, but not when absent. Results from a control experiment, which employed amodal variants of the stimuli, suggested that sounds impact the perceived brightness of the IC rather than shape formation per se. We provide the first demonstration that multisensory processes augment mid-level vision and everyday visual completion processes, and that one of the mechanisms is brightness enhancement. These results have important implications for the design of treatments and/or visual aids for low-vision patients.

Selective attention to sound features mediates cross-modal activation of visual cortices

“…It has been shown that sounds: i) improve the detection and discrimination of concurrently presented visual stimuli (e.g. McDonald et al, 2000; Gleiss and Kayser, 2014; Kayser and Kayser, 2018), ii) increase their subjective intensity (Stormer et al, 2009; Odgaard et al, 2004; Lovelace et al, 2003), iii) speed up responses to visual targets and enhance their attentional selection (Brang et al, 2015; Cappe et al, 2010; Matusz & Eimer, 2011, 2013; Matusz et al 2015a; 2019a, 2019b; also Kayser et al, 2017), and iv) even contribute to visual perceptual filling-in (Tivadar et al, 2018).…”

Auditory enhancement of illusory contour perception

“…The present results lend further support to the notion that sounds can enhance the excitability of visual cortices. In our prior work (Tivadar et al, 2018), we observed that an event-related potential signature of illusory contour sensitivity was enhanced by the presence of task-irrelevant sounds. A network of brain regions involved in this effect included not only LOC and the intra-parietal lobule, as previously observed in ERP studies of IC sensitivity (Murray & Herrmann, 2013), but also V1.…”

“…Instead, there was a general reduction in hit rates under multisensory versus unisensory conditions. Tivadar et al (2018) observed electrophysiological enhancement of event-related potential indices of illusory contour sensitivity despite near-ceiling performance and thus the absence of any concomitant behavioural effects when the task involved discrimination of IC presence versus absence.…”

“…A power analysis using G*Power (version 3.1.9.2) (Faul, Erdfelder, Lang, & Buchner, 2007) indicated that a sample size of 15 was necessary to obtain a statistical power of at least 0.95 for detecting an effect of a moderate size (f = 0.40) with our 2×2 factorial design and assuming a correlation of 0.50 between repeated measures. We also chose this sample size because our previous studies have obtained highly consistent data with similar sample sizes (Tivadar et al, 2018).…”

“…By contrast, it remains largely unknown whether multisensory processes would impact the processing of ICs at a behavioural level. Given recent evidence from our group indicating that neural mechanisms of IC perception can be impacted by task-irrelevant sounds (Tivadar, Retsa, Turoman, Matusz, & Murray, 2018), we here wanted to investigate behavioural effects of this enhancement.…”