Crossmodal correspondences: A tutorial review

Abstract

In many everyday situations, our senses are bombarded by many different unisensory signals at any given time. To gain the most veridical, and least variable, estimate of environmental stimuli/properties, we need to combine the individual noisy unisensory perceptual estimates that refer to the same object, while keeping those estimates belonging to different objects or events separate. How, though, does the brain "know" which stimuli to combine? Traditionally, researchers interested in the crossmodal binding problem have focused on the roles that spatial and temporal factors play in modulating multisensory integration. However, crossmodal correspondences between various unisensory features (such as between auditory pitch and visual size) may provide yet another important means of constraining the crossmodal binding problem. A large body of research now shows that people exhibit consistent crossmodal correspondences between many stimulus features in different sensory modalities. For example, people consistently match high-pitched sounds with small, bright objects that are located high up in space. The literature reviewed here supports the view that crossmodal correspondences need to be considered alongside semantic and spatiotemporal congruency, among the key constraints that help our brains solve the crossmodal binding problem.

The development of spontaneous sound-shape matching in monolingual and bilingual infants during the first year.

“…While the bouba-kiki effect is detected in infants around 11-adults (e.g., Maurer et al, 2006;Imai et al, 2008;Imai & Kita, 2014;Spence, 2011;etc. ), young infants only show sensitivities to arbitrary speech-sound correspondences before 6 months of age under specific circumstances (Experiment 1 of this study; Fort et al, 2013;Ozturk et al, 2013 when using non-words with syllable repetitions).…”

“…It is established that children and adults are sensitive to non-arbitrary crossmodal correspondences (for review, see Spence, 2011). A well-known example of this research is the tendency in adults and children to associate rounded shapes with specific types of speech sound combinations (e.g., "bouba"), but angular shapes with other types of speech sounds (e.g., "kiki").…”

Three-month-old human infants use vocal cues of body size

“…That is, motion paired with emitted sounds may satisfy the input conditions of adaptations for picking up on invariant sound/object co-relations, irrespective of whether the object is a living thing vocalizing, or a non-living object producing noise [27][28][29][30][31][32][33][34]54]. However, because the acoustics produced by the living and non-living worlds are interestingly different from one another, both in terms of their properties and also in their affordances [9,16,22,23], the psychological mechanisms for representing and reasoning about the living versus the non-living worlds will likely become increasingly distinct from one another over the course of development.…”

Generative Theatre of Totality

“…There are several studies regarding cross-modal correspondences in perception, and Spence's comprehensive tutorial highlights some of them (Spence, 2011). For example, high-pitched sounds are usually related to small bright lights and to higher spatial positioning, whereas slow movement is associated to darker ambiances, long and low pitched sounds.…”