Learning emotion-based acoustic features with deep belief networks

Abstract

The medium of music has evolved specifically for the expression of emotions, and it is natural for us to organize music in terms of its emotional associations. But while such organization is a natural process for humans, quantifying it empirically proves to be a very difficult task, and as such no dominant feature representation for music emotion recognition has yet emerged. Much of the difficulty in developing emotion-based features is the ambiguity of the ground-truth. Even using the smallest time window, opinions on the emotion are bound to vary and reflect some disagreement between listeners. In previous work, we have modeled human response labels to music in the arousal-valence (A-V) representation of affect as a time-varying, stochastic distribution. Current methods for automatic detection of emotion in music seek performance increases by combining several feature domains (e.g. loudness, timbre, harmony, rhythm). Such work has focused largely in dimensionality reduction for minor classification performance gains, but has provided little insight into the relationship between audio and emotional associations. In this new work we seek to employ regression-based deep belief networks to learn features directly from magnitude spectra. While the system is applied to the specific problem of music emotion recognition, it could be easily applied to any regression-based audio feature learning problem.

Learning Salient Features for Speech Emotion <newline/>Recognition Using Convolutional <newline/>Neural Networks

“…For SER, Stuhlsatz et al [36] used generatively pre-trained artificial neural networks (ANNs) to learn discriminative features of low dimension and found improvement in both weighted and unweighted recall on multiple emotion corpora. Schmidt and Kim [33] used deep belief networks (DBNs) to learn high-level features directly from magnitude spectra and achieved good performances on emotional music recognition compared to other feature extraction schemes. More recently, Le et al [17] investigated dynamic frame-level modeling with hybrid DBN-HMM classifiers on the FAU Aibo spontaneous emotion corpus [35] and achieved state-of-the-art results on the 5-class problem.…”

Deep Learning Our Everyday Emotions

“…Further examples for emotional speech recognition include [ [7,25,26,1,21,29]]. In a related way, deep learning has also been successfully applied to emotion recognition in music [ [31]]. Further, a number of works combine video cues such as facial information with speech analysis in a deep learning paradigm, such as in [ [24]] or in the winning contribution to the 2013 Emotion in the Wild Challenge held at ACM ICMI [ [23]] which was able to raise the organisers' baseline of 27.5 % accuracy to 41.0 %.…”

Bridge the semantic gap between pop music acoustic feature and emotion: Build an interpretable model

“…This brings a great challenge to describe and determine the goodness of each layer representation. Schmidt et al [15,16] used Deep Belief Networks (DBN) [17] with three hidden layers to learn emotion-based acoustic representations directly from magnitude spectra. Then, they obtained layer two of the DBN that performs best in terms of mean error.…”