Semantic Crosstalk in Timbre Perception

Abstract

Many adjectives for musical timbre reflect cross-modal correspondence, particularly with vision and touch (e.g., "darkbright," "smooth-rough"). Although multisensory integration between visual/tactile processing and hearing has been demonstrated for pitch and loudness, timbre is not well understood as a locus of cross-modal mappings. Are people consistent in these semantic associations? Do cross-modal terms reflect dimensional interactions in timbre processing? Here I designed two experiments to investigate crosstalk between timbre semantics and perception through the use of Stroop-type speeded classification. Experiment 1 found that incongruent pairings of instrument timbres and written names caused significant Stroop-type interference relative to congruent pairs, indicating bidirectional crosstalk between semantic and auditory modalities. Pre-Experiment 2 asked participants to rate natural and synthesized timbres on semantic differential scales capturing luminance (brightness) and texture (roughness) associations, finding substantial consistency for a number of timbres. Acoustic correlates of these associations were also assessed, indicating an important role for high-frequency energy in the intensity of cross-modal ratings. Experiment 2 used timbre adjectives and sound stimuli validated in the previous experiment in two variants of a semantic-auditory Stroop-type task. Results of linear mixed-effects modeling of reaction time and accuracy showed slight interference in semantic processing when adjectives were paired with cross-modally incongruent instrument timbres (e.g., the word "smooth" with a "rough" timbre). Taken together, I conclude by suggesting that semantic crosstalk in timbre processing may be partially automatic and could reflect weak synesthetic congruency between interconnected sensory domains.

The Communication of Timbral Intentions Between Pianists and Listeners and Its Dependence on Auditory-Visual Conditions

“…Timbre communication may also involve CMC features (Wallmark and Kendall, 2018 ; Wallmark, 2019a , b ). For example, timbre metaphors such as bright/dark, rough/smooth reflected cross-modal correspondence with vision and touch (Wallmark, 2019b ).…”

“…Timbre communication may also involve CMC features (Wallmark and Kendall, 2018 ; Wallmark, 2019a , b ). For example, timbre metaphors such as bright/dark, rough/smooth reflected cross-modal correspondence with vision and touch (Wallmark, 2019b ). Timbre metaphors in CMC categories encompass “an embodied conceptual transfer process by which an auditory target domain (timbre) is understood in reference to a non-auditory source domain (vision, touch, taste, and smell)” (2019a, p. 594).…”

Trilled /r/ is associated with roughness, linking sound and touch across spoken languages

“…Audiotactile interactions have also been shown for roughness perception 19 , and people can reliably judge textural roughness from acoustic cues alone 25 . Moreover, people readily classify sounds in terms of ‘roughness’ 26 – 28 . In English, the words rough and smooth are two very common descriptors of auditory qualities 29 , and, more generally, touch words are amongst the most frequent adjectives used to describe sound 30 – 32 .…”

Assessing the Role of Emotional Mediation in Explaining Crossmodal Correspondences Involving Musical Stimuli

“…Composed music is typically made up of a variety of different elements, including different musical notes, timbres, and vocal components, etc. As such, it is, a priori, uncertain as to whether people will preferentially experience, or establish, crossmodal correspondences with either the music's basic sensory features, such as its average pitch range, loudness, or timbre (Wallmark, 2019), or with more complex parameters such as, for example, the music's orchestration, melodic structure, tempo, or attack (see Cowles, 1935;Hubbard, 1996). It is an interesting empirical question, therefore, to consider which attribute(s) of a given musical excerpt may end up dominating in this regard.…”