At the Border of Acoustics and Linguistics: Bag-of-Audio-Words for the Recognition of Emotions in Speech

Abstract

Recognition of natural emotion in speech is a challenging task. Different methods have been proposed to tackle this complex task, such as acoustic feature brute-forcing or even endto-end learning. Recently, bag-of-audio-words (BoAW) representations of acoustic low-level descriptors (LLDs) have been employed successfully in the domain of acoustic event classification and other audio recognition tasks. In this approach, feature vectors of acoustic LLDs are quantised according to a learnt codebook of audio words. Then, a histogram of the occurring 'words' is built. Despite their massive potential, BoAW have not been thoroughly studied in emotion recognition. Here, we propose a method using BoAW created only of mel-frequency cepstral coefficients (MFCCs). Support vector regression is then used to predict emotion continuously in time and value, such as in the dimensions arousal and valence. We compare this approach with the computation of functionals based on the MFCCs and perform extensive evaluations on the RECOLA database, which features spontaneous and natural emotions. Results show that, BoAW representation of MFCCs does not only perform significantly better than functionals, but also outperforms by far most of recently published deep learning approaches, including convolutional and recurrent networks.

Automatic Detection of Major Depressive Disorder via a Bag-of-Behaviour-Words Approach

“…The bag-of-words (BoW) approach is known from natural language processing [30], which can be referred to the early description in [12]. Particularly, in the application of speech emotion recognition [21,26], and the area of health care [13,25], the BoAW approach achieved numerous excellent results. Motivated by the success of the BoAW method in aforementioned studies, we propose the badof-behaviour-words (BoBW) approach.…”

AVEC 2018 Workshop and Challenge

“…Emotion recognition from audiovisual signals usually relies on feature sets whose extraction is based on expertise gained over several decades of research in the domains of speech processing, e. g., Mel Frequency Cepstral Coefficients (MFCCs), and vision computing, e. g., Facial Action Units (FAUs). However, recent advances in the field of representation learning, whose objective is to learn representations of data that are best suited for the recognition task [6], have shown that efficient representations of audiovisual signals can be learnt in the context of emotion [2,59,71].…”

Spotting Social Signals in Conversational Speech over IP: A Deep Learning Perspective

“…Further, we will look into the data imbalance effects of the database and how this could possibly improve robustness. Moreover, we will combine LSTM and GRU networks on the recently proposed Bag-Of-AudioWords approach [30]. Finally, we also plan to do a full endto-end training of the combined feature and posterior models and examine other network architectures, such as variants of the LSTM models or Convolutional Neural Networks.…”