No abstract

A series of four experiments explored how cross-modal similarities between sensory attributes in vision and hearing reveal themselves in speeded, two-stimulus discrimination. When subjects responded differentially to stimuli on one modality, speed and accuracy of response were greater on trials accompanied by informationally irrelevant "matching" versus "mismatching" stimuli from the other modality. Cross-modal interactions appeared in (a) responses to dim/bright lights and to dark/light colors accompanied by low-pitched/high-pitched tones; (b) responses to low-pitched/high-pitched tones accompanied by dim/bright lights or by dark/light colors; (c) responses to dim/bright lights, but not to dark/light colors, accompanied by soft/loud sounds; and (d) responses to rounded/sharp forms accompanied by low-pitched/high-pitched tones. These results concur with findings on cross-modal perception, synesthesia, and synesthetic metaphor, which reveal similarities between pitch and brightness, pitch and lightness, loudness and brightness, and pitch and form. The cross-modal interactions in response speed and accuracy may take place at a sensory/perceptual level of processing or after sensory stimuli are encoded semantically.

Presented with a tone pitched at SO cycles per second and an amplitude of 100 decibels, S. saw a brown strip against a dark background that had red, tonguelike edges. The sense of taste he experienced was like that of sweet and sour borscht, a sensation that gripped his entire tongue. . . . Presented with a tone pitched at 2,000 cycles per second and having an amplitude of 113 decibels, S. said: "It looks something like fireworks tinged with a pink-red hue. The strip of color feels rough and unpleasant, and it has an ugly taste-rather like that of a briny pickle." (pp. 45-46

In two experiments, we examined whether or not pairs of auditory dimensions-timbre-loudness (Experiment 1) and timbre-pitch (Experiment 2)-interact in speeded classification. Subjects classified values from one dimension while the other dimension was (1) held constant (baseline), (2) varied orthogonally (filtering), or (3) correlated linearly. The subjects showed substantial Garner interference when classifying all dimensions-that is, poor performance at filtering relative to baseline. Timbre and loudness displayed redundancy gain (i.e., performance faster than baseline) when correlated positively, but redundancy loss (i.e., interference) when correlated negatively. Timbre and pitch displayed redundancy gain however dimensions were correlated. Both pairs of dimensions showed substantial effects of congruity: Attributes from one dimension were classified faster when paired with "congruent" attributes from the other dimension. The results are interpreted in terms of an interactive multichannel model of auditory processing.