Audiovisual correspondence between musical timbre and visual shapes

Abstract

This article investigates the cross-modal correspondences between musical timbre and shapes. Previously, such features as pitch, loudness, light intensity, visual size, and color characteristics have mostly been used in studies of audio-visual correspondences. Moreover, in most studies, simple stimuli e.g., simple tones have been utilized. In this experiment, 23 musical sounds varying in fundamental frequency and timbre but fixed in loudness were used. Each sound was presented once against colored shapes and once against grayscale shapes. Subjects had to select the visual equivalent of a given sound i.e., its shape, color (or grayscale) and vertical position. This scenario permitted studying the associations between normalized timbre and visual shapes as well as some of the previous findings for more complex stimuli. One hundred and nineteen subjects (31 females and 88 males) participated in the online experiment. Subjects included 36 claimed professional musicians, 47 claimed amateur musicians, and 36 claimed non-musicians. Thirty-one subjects have also claimed to have synesthesia-like experiences. A strong association between timbre of envelope normalized sounds and visual shapes was observed. Subjects have strongly associated soft timbres with blue, green or light gray rounded shapes, harsh timbres with red, yellow or dark gray sharp angular shapes and timbres having elements of softness and harshness together with a mixture of the two previous shapes. Color or grayscale had no effect on timbre-shape associations. Fundamental frequency was not associated with height, grayscale or color. The significant correspondence between timbre and shape revealed by the present work allows designing substitution systems which might help the blind to perceive shapes through timbre.

A Flexible Bio-Inspired Hierarchical Model for Analyzing Musical Timbre

“…We evaluated the proposed model in three different applications to demonstrate its potential as a general framework for timbre representation: 1) evaluating roughness extraction and comparing results with subjective data from a previous psychoacoustic study [35] 2) musical instrument classification using k-NN and a Bayesian network and 3) selection of timbral features that best represent the sounds that had been labeled in an audio-visual experiment [36].…”

“…The idea was to use the model to find the best features that could distinguish between three timbral classes consisting of 23 sounds. We used these sounds in an earlier audio-visual experiment, where 119 subjects were asked to select a visual shape out of a set of three for each [36]. Results showed that there existed a strong correspondence between timbre and visual shapes.…”

“…All the 23 sounds used in [36] were 1 second in duration and were sampled at 44100 Hz. They included notes from piano, cello, guitar, marimba and saxophone all of which had been multiplied by a Hanning window with a duration of 1 second to equalize the onset and offset such that only the spectral features of timbre were preserved.…”

“…Sounds associated with the visual shapes labeled as S 1 , S 2 , and, S 3 in [36] are referred to as class 1, class 2, and class 3, respectively. Sounds derived from crash cymbals, gong and triangle belong to class 1.…”

“…The log attack time and spectral centroid were found to be the best descriptors for the three timbral classes. The 3 clusters obtained by the k-means algorithms for the 3 timbral classes of [36] are shown. The relative positioning of the three timbral classes in this space is in accordance with the perceptual associations obtained in [36].…”