The role of visual experience in the emergence of cross-modal correspondences

Abstract

Cross-modal correspondences describe the widespread tendency for attributes in one sensory modality to be consistently matched to those in another modality. For example, high pitched sounds tend to be matched to spiky shapes, small sizes, and high elevations. However, the extent to which these correspondences depend on sensory experience (e.g. regularities in the perceived environment) remains controversial. Two recent studies involving blind participants have argued that visual experience is necessary for the emergence of correspondences, wherein such correspondences were present (although attenuated) in late blind individuals but absent in the early blind. Here, using a similar approach and a large sample of early and late blind participants (N = 59) and sighted controls (N = 63), we challenge this view. Examining five auditory-tactile correspondences, we show that only one requires visual experience to emerge (pitch-shape), two are independent of visual experience (pitch-size, pitch-weight), and two appear to emerge in response to blindness (pitch-texture, pitch-softness). These effects tended to be more pronounced in the early blind than late blind group, and the duration of vision loss among the late blind did not mediate the strength of these correspondences. Our results suggest that altered sensory input can affect cross-modal correspondences in a more complex manner than previously thought and cannot solely be explained by a reduction in visually-mediated environmental correlations. We propose roles of visual calibration, neuroplasticity and structurally-innate associations in accounting for our findings.

Heaviness-brightness correspondence and stimulus-response compatibility

“…The present study confirms that heaviness contributes to cross-sensory correspondences in its own right, and not simply through its association with size. Together with evidence from other studies (e.g., Hamilton-Fletcher et al, 2018;, size and heaviness are confirmed as being dissociable feature dimensions, each enjoying its own correspondences. This is an important outcome because it helps to validate the identities of the conceptual dimensions involved in cross-sensory correspondences.…”

“…Especially persuasive evidence against collapsing size and heaviness as cross-sensory feature dimensions is provided by Hamilton-Fletcher et al (2018). As in the present study, their participants were presented with pairs of objects to hold in their hands, where the objects in each pair had been carefully created to contrast in just a single feature.…”

"Like Popcorn"

“…The authors found that participants who used the SSD were able to discriminate shapes through sound after only minimal training. Our findings could extend this research and other work on crossmodal display for accessible technology in the design of SSDs [41,85], for example by augmenting audio-based SSDs with olfactory and tactile displays, and also contribute to the design of accessible technologies more broadly.…”

“…Finally, this work also contributes to the growing body of research on crossmodal correspondences in the sighted and visually impaired [27,41], and we further contribute new accessible research methods, such as the verbal Self-Assessment Manikin (SAM), tactile "Bouba" and "Kiki", and 3D printed slider with haptic and audio feedback, which can facilitate the inclusion of visually impaired participants in future crossmodal research.…”

Sound symbolism in sighted and blind. The role of vision and orthography in sound-shape correspondences

“…Alternatively, early blindness can impact directly on some specific sound-symbolic mechanisms that remain unspecified in our analysis. For instance, a recent paper published while our paper was under revision, showed that, contrary to sighted, early blind (n= 32) failed to associate high-pitch tones with jagged shapes and low-pitch tones with round shapes (Hamilton-Fletcher et al, 2018). Indeed, there is a large literature suggesting that low pitch is associated with larger/darker/lower/rounded characteristics of objects and events Spence, 2011), and to the extent that sound-shape correspondences such as the bouba-kiki effect are based on pitch-shape associations (Chandran, Banerjee, & Ghosh, 2017;Nielsen & Rendall, 2011;, they should be different between sighted and early blind.…”