Multistage linguistic conditioning of convolutional layers for speech emotion recognition

Abstract

In this contribution, we investigate the effectiveness of deep fusion of text and audio features for categorical and dimensional speech emotion recognition (SER). We propose a novel, multistage fusion method where the two information streams are integrated in several layers of a deep neural network (DNN), and contrast it with a single-stage one where the streams are merged in a single point. Both methods depend on extracting summary linguistic embeddings from a pre-trained BERT model, and conditioning one or more intermediate representations of a convolutional model operating on log-Mel spectrograms. Experiments on the widely used IEMOCAP and MSP-Podcast databases demonstrate that the two fusion methods clearly outperform a shallow (late) fusion baseline and their unimodal constituents, both in terms of quantitative performance and qualitative behaviour. Our accompanying analysis further reveals a hitherto unexplored role of the underlying dialogue acts on unimodal and bimodal SER, with different models showing a biased behaviour across different acts. Overall, our multistage fusion shows better quantitative performance, surpassing all alternatives on most of our evaluations. This illustrates the potential of multistage fusion in better assimilating text and audio information.

Dawn of the transformer era in speech emotion recognition: closing the valence gap

“…The numbers back up two of our earlier findings: the large architecture is superior to the base model and HuBERT outperforms wav2vec 2.0. Their CCC performance surpasses both that of Triantafyllopoulos et al [4] (.515), who proposed a multimodal fusion of pre-trained BERT embeddings with an untrained CNN model, and of Li et al [35] (.377) who pre-train a CRNN model on LibriSpeech using Contrastive Predictive Coding and subsequently fine-tuned it on MSP-Podcast.…”

“…Inspired by Wang et al [23], we use a simple head architecture, which we build on top of wav2vec 2.0 [21] or HuBERT [22] (see Figure 1): we apply average pooling over the hidden states of the last transformer layer and feed the result through a hidden layer and a final output layer (the pooled embeddings and the hidden layer outputs are dropped out). For fine-tuning on the downstream task, we use the ADAM optimiser with CCC loss, which is the standard loss function used for dimensional SER [4,35,43], and a fixed learning rate of 1e−4. We run for 5 epochs with a batch size of 32 and keep the checkpoint with best performance on the development set.…”

“…The goal of an automatic SER system is to analyse the voice signal and derive a prediction of an emotional category or dimensional value. This can be done either through the linguistic (what has been said) or the paralinguistic (how it has been said) stream [4] -or both. The former is primarily contained in textual information (e. g. the transcription of an input utterance) while the latter in the acoustic and prosodic information contained in the raw audio signal.…”

“…Each stream comes with its pros and cons: linguistics is better suited for valence recognition [5,6,7,8] and can heavily draw from recent advances in automatic speech recognition (ASR) and natural language processing (NLP) [9,10,11], but might be limited to a single language. Paralinguistics works better for arousal and dominance [5,6,7,8] and has the potential to generalise across different languages, while typically suffering from low valence performance [4,8]. Of course, the strengths of both paradigms can be utilised in complementary fashion in multimodal 1 architectures.…”

“…Although this field has seen tremendous progress in the last decades [1], three major challenges remain for real-world paralinguistics-based SER applications: a) improving on its inferior valence performance [4,8], b) overcoming issues of generalisation and robustness [12,13], and c) alleviating individual-and group-level fairness concerns, which is a prerequisite for ethical emotion recognition technology [14,15]. Previous works have attempted to tackle these issues in isolation, e. g. by using cross-modal knowledge distillation to increase valence performance [16], speech enhancement or data augmentation to improve robustness [12,13], and de-biasing techniques to mitigate unfair outcomes [17].…”