An extended research of crossmodal correspondence between color and sound in psychology and cognitive ergonomics

Abstract

Based on the existing research on sound symbolism and crossmodal correspondence, this study proposed an extended research on cross-modal correspondence between various sound attributes and color properties in a group of non-synesthetes. In Experiment 1, we assessed the associations between each property of sounds and colors. Twenty sounds with five auditory properties (pitch, roughness, sharpness, tempo and discontinuity), each varied in four levels, were used as the sound stimuli. Forty-nine colors with different hues, saturation and brightness were used to match to those sounds. Result revealed that besides pitch and tempo, roughness and sharpness also played roles in sound-color correspondence. Reaction times of sound-hue were a little longer than the reaction times of sound-lightness. In Experiment 2, a speeded target discrimination task was used to assess whether the associations between sound attributes and color properties could invoke natural cross-modal correspondence and improve participants’ cognitive efficiency in cognitive tasks. Several typical sound-color pairings were selected according to the results of Experiment 1. Participants were divided into two groups (congruent and incongruent). In each trial participants had to judge whether the presented color could appropriately be associated with the sound stimuli. Result revealed that participants responded more quickly and accurately in the congruent group than in the incongruent group. It was also found that there was no significant difference in reaction times and error rates between sound-hue and sound-lightness. The results of Experiment 1 and 2 indicate the existence of a robust crossmodal correspondence between multiple attributes of sound and color, which also has strong influence on cognitive tasks. The inconsistency of the reaction times between sound-hue and sound-lightness in Experiment 1 and 2 is probably owing to the difference in experimental protocol, which indicates that the complexity of experiment design may be an important factor in crossmodal correspondence phenomena.

QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate

“…The findings also indicate that the positive impact of multisensorial effects on users' QoE is substantiated by integrating cross-modally mapped component effects in a mulsemedia context. This implies there exists a noticeable cross-modal correspondence in a digital world between the visual features of videos and audio pitches which substantiates studies in [19,61,69]. Similarly, such correspondence exists between the visual features of the videos and smell effects [19,21,61,64,69].…”

“…The accompanying auditory and olfactory content was modified in line with principles of auditory-visual and olfactory-visual crossmodal correspondences that were previously shown in the literature. The video with dominant yellow images (V1) was watched accompanied by high pitch sounds and bergamot odor, while the one dominantly blue (V2) by low pitch sounds and lilial odor [19,61,69].…”

Using Eye Tracking and Heart-Rate Activity to Examine Crossmodal Correspondences QoE in Mulsemedia

“…The accompanying sound was modified to respect the principles of auditory-visual crossmodal correspondences as demonstrated in the literature. The video with dominant yellow images (V1) was watched accompanied by high pitch sounds, while the one dominantly blue (V2) by low pitch sounds [Simpson et al 1956;Sun et al 2018]. Then, when brightness was considered the dominant visual characteristic, low pitch sounds were used for the video with low level of brightness (V3), while the one with high brightness values contained high pitch (V4) sounds, based on [Marks 1974].…”

The effect of sound environment on spatial knowledge acquisition in a virtual outpatient polyclinic

“…There is a long history of research on cross-modal correspondence between auditory stimuli and color (Cytowic, 2002;Menouti et al, 2015;Sun et al, 2018;Ward et al, 2006). People tend to make reliable associations between specific dimensions of vision and sounds, such as loudness, visual size, brightness, and hue.…”