Steady state visual evoked potentials reveal a signature of the pitch-size crossmodal association in visual cortex

Abstract

Crossmodal correspondences describe our tendency to associate sensory features from different modalities with each other, such as the pitch of a sound with a specific size of a visual object. While such crossmodal correspondences (or associations) have been described in many behavioural studies their neurophysiological correlates remain debated. Under the current working model of multisensory perception both a low- and a high-level account seem plausible. That is, the neurophysiological processes shaping these associations could commence in early sensory regions, or may rather emerge only later and as a result from high-level multisensory integration processes or in semantic and object identification networks. We implemented a paradigm based on steady-state visual evoked potentials (SSVEP) to directly test the hypothesis that the associations between acoustic pitch and the visual features of size, hue or chromatic saturation are visible in brain activity from early visual cortices. Our results reveal a pitch-size congruency effect in the SSVEP over occipital electrodes and in the behavioural data, supporting a low-level account of this crossmodal association. For the associations between pitch and chromatic features we found no significant effects. We speculate that this signature of the pitch size association in early visual cortices reflects the successful pairing of congruent visual and acoustic object properties and may contribute to establishing causal relations between multisensory objects.

Involvement of the superior colliculi in crossmodal correspondences

“…Finally, a study by Sciortino and Kayser (2023) looked for integration of pitch with size, hue, and chromatic saturation in the primary visual cortex using steady state visual evoked potentials (SSVEP). The authors were motivated in part by the previous conflicting findings in CMC EEG data.…”