Sonification of Emotion: Strategies and results from the intersection with music

Abstract

Emotion is a word not often heard in sonification, though advances in affective computing make the data type imminent. At times the relationship between emotion and sonification has been contentious due to an implied overlap with music. This paper clarifies the relationship, demonstrating how it can be mutually beneficial. After identifying contexts favourable to auditory display of emotion, and the utility of its development to research in musical emotion, the current state of the field is addressed, reiterating the necessary conditions for sound to qualify as a sonification of emotion. With this framework, strategies for display are presented that use acoustic and structural cues designed to target select auditory-cognitive mechanisms of musical emotion. Two sonifications are then described using these strategies to convey arousal and valence though differing in design methodology: one designed ecologically, the other computationally. Each model is sampled at 15-second intervals at 49 evenly distributed points on the AV space, and evaluated using a publically available tool for computational music emotion recognition. The computational design performed 65 times better in this test, but the ecological design is argued to be more useful for emotional communication. Conscious of these limitations, computational design and evaluation is supported for future development.

Interactive sonification strategies for the motion and emotion of dance performances

“…In other words, choosing to sacrifice data granularity in exchange for communication efficiency is common practice in both domains of sonification and visualization. This amount of control can aid in framing the designer's intended message, but it can also obfuscate trends in the data if designed poorly (Winters & Wanderley, 2014).…”

“…Detecting the presence of affective information and transferring emotion to another involve two separate but related cognitive mechanism (Winters & Wanderley, 2014). This dissertation is primarily concerned with how composers/choreographers code and how listeners/viewers decode affective information in music and dance.…”

“…In cultures with western music traditions, children as young as four have learned through musical exposure to associate the harmony of a major chord with positive valence, and the dissonance of a minor chord with negative valence (Juslin & Laukka, 2003). Harmony and dissonance can represent a continuum of valence, providing more precision than the discrete states of melody contour direction, or tonal key (Winters & Wanderley, 2014). In standard western music theory, each pitch interval has a specific harmonic function within a scale that contributes to subjective feelings of melodic tension, stability, and resolution (Bharucha & Krumhansl, 1983;Bigand, Parncutt, & Lerdahl, 1996).…”

“…Embedding affective information in displays has received a relatively small amount of attention in the sonification literature, despite the close relation between music and emotion (Winters & Wanderley, 2014). Sonification, however, is well suited at making small changes in a continuous variable perceptible (Hermann et al, 2011), including affective information (Winters & Wanderley, 2013).…”

Interactive Sonification Strategies for the Motion and Emotion of Dance Performances

“…The sonification of movement data has shown promising application in the domains of athletic training (Schaffert, Mattes, Barrass, & Effenberg, 2009), physical rehabilitation Danna et al, 2013), and artistic installations (Camurri, De Poli, Friberg, Leman, & Volpe, 2005). The sonification of emotion is less explored in the sonification literature save for a few examples (Friberg, 2006;Winters & Wanderley, 2014). Fortunately, there is a large amount of research on emotion perception from other academic fields (Ekman, 2016;Gabrielsson & Juslin, 1996;Jeon, 2017;Juslin, 2000;Juslin & Laukka, 2003;Sterkenburg, Jeon, & Plummer, 2014;Williams, Kirke, Miranda, Roesch, & Nasuto, 2013) to help inform the design of a novel framework for the systematic sonification of emotion.…”