A Common Scheme for Cross-Sensory Correspondences across Stimulus Domains

Abstract

Following Karwoski, Odbert, and Osgood (1942, Journal of General Psychology 26 199-222), it is proposed that cross-sensory correspondences can arise from extensive, bi-directional cross-activation between dimensions of connotative meaning. If this account is correct, the same set of cross-sensory correspondences (eg brightness with high pitch, high pitch with sharpness, smallness with brightness) should emerge regardless of the sensory channel (auditory, visual, or tactile) that is probed. To test this prediction, participants rated a range of auditory, visual, and tactile stimuli on a series of rating scales relating to different dimensions of connotative meaning. With only a few minor exceptions, the same set of cross-sensory correspondences emerged from all types of stimulus variation. This supports the suggestion that cross-sensory correspondences can reflect reciprocal interactions between dimensions of connotative meaning, and indicates that Spence's (2011, Attention, Perception, & Psychophysics 73 971-995) theoretical framework might be usefully extended to include semantically-based correspondences.

Contingent sounds change the mental representation of one’s finger length

“…Recent evidence shows that a sound with rising pitch not only corresponds to an upward visual movement, but also to an upward tactile movement 28 (see also the report of a static tone/upper tactile location correspondence 29 ). Besides this correspondence with height, pitch is also associated to physical size in the visual domain, with static high and low pitches being respectively congruent with smaller and larger visual size 23,30–33 , and ascending and descending pitches being respectively congruent with growing and shrinking size 34 .…”

The role of conceptual knowledge in understanding synaesthesia: Evaluating contemporary findings from a “hub-and-spokes” perspective

“…We further speculate that such shared mechanisms operate at the stage of conceptual processing: there is accumulating evidence that cross-modal correspondences rely more on cognitive processing at an abstract conceptual level and perhaps less on mechanisms of the perceptual system. For instance, the mapping between pitch and location has been demonstrated to rely on recognition of a sound’s relative pitch height within a given context rather than acoustic feature of absolute pitch height (Chiou and Rich, 2012); the mapping between pitch and size has been found to be semantically mediated (Gallace and Spence, 2006); and the same set of conceptual regularities seem to guide whether various sensory attributes form a harmonious combination or not (Walker, 2012; Walker and Walker, 2012; Walker et al, 2012). Furthermore, the impact of some cross-modal mappings (e.g., pitch-size; loudness–brightness) on perceptual judgment has been shown to reflect biases at a post-perceptual decisional level, rather than perceptual-level phenomena (Marks et al, 2003; Keetels and Vroomen, 2011).…”

Newborns are sensitive to the correspondence between auditory pitch and visuospatial elevation

“…Similar cross‐sensory mappings involving pitch appear in the sound‐induced visual imagery experienced by visual‐hearing synaesthetes (Chiou, Stelter, & Rich, ; Ward, Huckstep, & Tsakanikos, ), and in the congruity effects observed during the speeded classification of elementary stimulus features (e.g., Evans & Treisman, ). Of course, correspondences are not confined to auditory‐visual mappings, but instead extend across all sensory domains, such as when seeing a relatively dark object induces expectations that it will be relatively heavy (Walker, ; Walker, Francis, & Walker, ; Walker, Walker & Francis, ), again despite there being many exceptions to this generalization.…”