On the Evolution of Speech Representations for Affective Computing: A brief history and critical overview

Abstract

ecent advances in the field of machine learning have shown great potential for the automatic recognition of apparent human emotions. In the era of Internet of Things (IoT) and big-data processing, where voice-based systems are well established, opportunities to leverage cutting-edge technologies to develop personalised and humancentered services are genuinely real, with a growing demand in many areas such as education, health, well-being and entertainment. Automatic emotion recognition from speech, which is a key element for developing personalised and human-centered services, has reached a degree of maturity that makes it of broad commercial interest today.However, there are still major limiting factors that prevent a broad applicability of emotion recognition technology. For example, one open challenge is the poor generalisation capabilities of currently used feature extraction techniques to interpret expressions of affect across different persons, contexts, cultures and languages.Since speech and emotion involve interdependent cognitive processes, emotion can be observed both in the spoken words and in the acoustic properties of the speech signal, where many other factors such as gender, age, culture and personality come into play. Even though features derived from speech science have permitted to describe and predict some expressions of affect relatively well, these representations do not encompass all the perceptual cues that humans may sense during an emotional experience. With the advancement of machine (deep) learning, computational methods have been proposed for learning representations from raw speech data. Newly introduced deep representations, although not as easily interpretable as most descriptors from speech science, promise to disentangle many existing issues in affective computing research, such as lack of labelled data, robustness to noise, and domain mismatch [1,2,3].In this contribution, we provide a brief history and critical overview of the different speech representations that have been used in automatic emotion recognition over the years (cf. Figure 1), focusing on how and why the new unsupervised representations in particular can provide major unprecedented benefits in affective computing.Thus, in here, we stay mainly on the topic of speech representations but also mention the new trend to integrate

End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning

“…Similarly, in the larger and more complex MSPConv, the E2E based BBB-KL model achieves the best uncertainty estimate performances, against all other models in comparison, with 0.1181 L ccc (s) and 0.3571 L KL . This trend is inline with literature which suggest end-to-end learning for uncertainty modeling [14].…”

“…Recently, end-to-end architectures have been shown to deliver state-of-the-art emotion predictions [11]- [13], by learning features rather than relying on hand-crafted features. For modeling subjectivity in emotions, it has been conjured that end-to-end learning also promotes learning subjectivity dependent representations [14].…”

Multimodal interaction enhanced representation learning for video emotion recognition

“…(Baltrusaitis et al, 2018 ). Alternatively, benefiting from the development of deep learning, deep-learned feature representations based on the large-scale pre-trained convolutional neural networks (CNN) such as ResNet (He et al, 2016 ) and VGGish (Hershey et al, 2017 ) also have been widely used for emotion recognition (Alisamir and Ringeval, 2021 ; Li and Deng, 2022 ). Compared with those hand-crafted features, the pre-trained CNN encoders can extract more powerful visual/audio features.…”

Multi-Corpus Affect Recognition with Emotion Embeddings and Self-Supervised Representations of Speech

“…Authors showed that W2V2 representations allow the use of less complex models, compared to MFB features, concluding that W2V2 representations provide contextualised information of speech that are robust in different contexts. Thus, by using self-supervised representations, less labelled data are needed for the downstream task, which is beneficial for SER, as it is highly susceptible to data scarcity issues [18].…”

“…Self-Supervised Learning (SSL) methods such as the contrastive loss objective used to build a W2V2 model [15], do not need any labels to learn contextualised abstractions of speech, and can thus benefit from the abundance of unlabelled data. Therefore, by training SSL models on large amounts of data, we can achieve highly contextualised representations of speech that are robust against domain mismatch issues [16]- [18]. The robustness against unseen data can also be further improved by training on data from several different domains [19].…”