On the Acoustics of Emotion in Audio: What Speech, Music, and Sound have in Common

Abstract

Without doubt, there is emotional information in almost any kind of sound received by humans every day: be it the affective state of a person transmitted by means of speech; the emotion intended by a composer while writing a musical piece, or conveyed by a musician while performing it; or the affective state connected to an acoustic event occurring in the environment, in the soundtrack of a movie, or in a radio play. In the field of affective computing, there is currently some loosely connected research concerning either of these phenomena, but a holistic computational model of affect in sound is still lacking. In turn, for tomorrow’s pervasive technical systems, including affective companions and robots, it is expected to be highly beneficial to understand the affective dimensions of “the sound that something makes,” in order to evaluate the system’s auditory environment and its own audio output. This article aims at a first step toward a holistic computational model: starting from standard acoustic feature extraction schemes in the domains of speech, music, and sound analysis, we interpret the worth of individual features across these three domains, considering four audio databases with observer annotations in the arousal and valence dimensions. In the results, we find that by selection of appropriate descriptors, cross-domain arousal, and valence regression is feasible achieving significant correlations with the observer annotations of up to 0.78 for arousal (training on sound and testing on enacted speech) and 0.60 for valence (training on enacted speech and testing on music). The high degree of cross-domain consistency in encoding the two main dimensions of affect may be attributable to the co-evolution of speech and music from multimodal affect bursts, including the integration of nature sounds for expressive effects.

Comparing the acoustic expression of emotion in the speaking and the singing voice

“…Another avenue for further development, especially when data for larger groups of singers and actors become available, is the use of machine learning to assess the degree of generalizability of the underlying acoustic profiles. A recent study by Weninger et al (2013) shows that algorithms trained on emotional music are quite successful on emotional speech and vice versa, suggesting that this may be a very viable approach. Weniger et al show that the effect generalizes, to some extent, even to environmental sounds.…”

Emotion in the singing voice—a deeperlook at acoustic features in the light ofautomatic classification

“…Previous findings in [18] suggest that the expression of emotions in speaking and singing voice are related. Further, [12] concludes that similar methods and acoustic features can be used to automatically classify emotions in speech, polyphonic music, as well as emotions perceived by listeners in or associated by them with other, general sounds. This suggests that the methods for speech emotion recognition can be transferred to singing emotion recognition.…”

“…[10][11][12]), although the fact that emotions are visible in acoustic properties of the voice has been frequently acknowledged [13,14]. In particular, in music, emotions play a major role and singers must be able to easily express a wide range of emotions.…”

The Conflict Escalation Resolution (CONFER) Database

“…In this work, we employ the openSMILE feature extractor [36] to obtain the ComParE acoustic feature set of 65 low-level descriptors (LLD) (4 energy-related, 55 spectral and 6 voicing-related), which has been successfully applied for automatic recognition of paralinguistic phenomena [37]. The 65 LLD used are summarized in Table 3 in [38].…”

“…The audio features extracted for each sequence of the CONFER Database are down-sampled to 25 Hz frequency to match the frame rate of the video stream. Similarly to [39,37] the audio features of each sequence are z-normalized (each feature component is normalized to mean=0 and standard deviation=1). Visual features.…”