Jointly Predicting Arousal, Valence and Dominance with Multi-Task Learning

Abstract

An appealing representation of emotions is the use of emotional attributes such as arousal (passive versus active), valence (negative versus positive) and dominance (weak versus strong). While previous studies have considered these dimensions as orthogonal descriptors to represent emotions, there are strong theoretical and practical evidences showing the interrelation between these emotional attributes. This observation suggests that predicting emotional attributes with a unified framework should outperform machine learning algorithms that separately predict each attribute. This study presents methods to jointly learn emotional attributes by exploiting their interdependencies. The framework relies on multi-task learning (MTL) implemented with deep neural networks (DNN) with shared hidden layers. The framework provides a principled approach to learn shared feature representations that maximize the performance of regression models. The results of within-corpus and cross-corpora evaluation show the benefits of MTL over single task learning (STL). MTL achieves gains on concordance correlation coefficient (CCC) as high as 4.7% for within-corpus evaluations, and 14.0% for cross-corpora evaluations. The visualization of the activations of the last hidden layers illustrates that MTL creates better feature representation. The best structure has shared layers followed by attribute-dependent layers, capturing better the relation between attributes.

Improving Cross-Corpus Speech Emotion Recognition with Adversarial Discriminative Domain Generalization (ADDoG)

“…Most modern techniques of cross-corpus speech emotion recognition use deep learning to build representations over lowlevel acoustic features. Many of these techniques incorporate tasks in addition to emotion to be able to learn more robust representations [6], [10].…”

Multitask Learning and Multistage Fusion for Dimensional Audiovisual Emotion Recognition

“…0.7, 0.2, and 1.0 for α, β, and γ, respectively. Our proposed MTL learning with three parameters outperforms STL and previous MTL [4]. For STL approaches, both arousal and valence obtained the highest CCC score when its attribute is optimized.…”

“…where α, β, and γ are the weighting factors for each emotion dimension loss function. In a common approach, α, β, and γ are set to be 1, while in [4], γ is set to be 1 − (α + β) to minimize MSE. In that approach, all weighting factors are in range 0-1.…”

Speech Emotion Recognition via Contrastive Loss under Siamese Networks

“…In the first experiment, we discuss the influence of different feature selection manners and prediction manners. Since multi-task learning shows its performance in [6,20,21], we treat the multi-task learning as the comparison approach in the second experiment. In the last experiment, we show the advantages of adding the contrastive loss during the training phase.…”