A multitask approach to continuous five-dimensional affect sensing in natural speech

Abstract

Automatic affect recognition is important for the ability of future technical system to interact with us socially in an intelligent way by understanding our current affective state. In recent years there has been a shift in the field of affect recognition from "in the lab" experiments with acted data to "in the wild" experiments with spontaneous and naturalistic data. Two major issues thereby are the proper segmentation of the input and adequate description and modelling of affective states. The first issue is crucial for responsive, real-time systems such as virtual agents and robots, where the latency of the analysis must be as small as possible. To address this issue we introduce a novel method of incremental segmentation to be used in combination with supra-segmental modelling For modelling of continuous affective states we use Long Short-Term Memory Recurrent Neural Networks, with which we can show an improvement in performance over standard recurrent neural networks and feed forward neural networks as well as Support Vector Regression. For experiments we use the SEMAINE database, which contains recordings of spontaneous and natural human to Wizard-of-Oz conversations. The recordings are annotated continuously in time and magnitude with FeelTrace for five affective dimensions, namely activation, expectation, intensity, power/dominance, and valence. To exploit dependencies between the five affective dimensions we investigate multi-task learning of all five dimensions augmented with inter-rater standard deviation. We can show improvements for multi-task over single task modelling. Correlation coefficients of up to 0.81 are obtained for the activation dimension and up to 0.58 for the valence dimension. The performance for the remaining dimensions were found to be in between that for activation and valence.

Continuous Estimation of Emotions in Speech by Dynamic Cooperative Speaker Models

“…Results show that arousal is significantly better recognised from the acoustic features than valence. This result is in agreement with the literature, where acoustic features have always been shown to present a stronger correlation with the arousal dimension in comparison with valence [21], [25], [26], [30], [37]. The values of CCC and CC are most of the time almost identical, as the RMSE is quite low; we obtained an average RMSE of 0.068 for arousal and of 0.128 for valence over a range of 2.…”

“…However, the natural diversity found in emotion perception is usually merged when a machine learning model is trained, by averaging several evaluations from a pool of raters into a single gold standard. Whereas the use of all annotation data can help at preserving diversity in emotion perception, e. g., by using multi-task learning of each annotator [25], [26], it has the main disadvantage to increase the overall complexity of the model according to the number of available raters. The issue of synchronisation of various individual ratings for defining a gold standard has also been investigated with signal processing techniques.…”

Attention-augmented End-to-end Multi-task Learning for Emotion Prediction from Speech

“…For speech emotion prediction, MTL has been frequently utilised. Eyben et al [17] firstly proposed to jointly train five different emotional dimensions for continuous emotion recognition. The experimental results have clearly indicated that the MTL model remarkably outperforms single-task-based models.…”

Modeling subjectiveness in emotion recognition with deep neural networks: Ensembles vs soft labels

“…It was reported that while ground truth hard labels performed better than soft labels, soft labels had a more similar entropy to human annotators. In [15], the inter-annotator standard deviation was used to model the variability between multiple annotators in a multi-task learning emotion recognition framework.…”