Predicting Categorical Emotions by Jointly Learning Primary and Secondary Emotions through Multitask Learning

Abstract

Detection of human emotions is an essential part of affect-aware human-computer interaction (HCI). In daily conversations, the preferred way of describing affects is by using categorical emotion labels (e.g., sad, anger, surprise). In categorical emotion classification, multiple descriptors (with different degrees of relevance) can be assigned to a sample. Perceptual evaluations have relied on primary and secondary emotions to capture the ambiguous nature of spontaneous recordings. Primary emotion is the most relevant category felt by the evaluator. Secondary emotions capture other emotional cues also conveyed in the stimulus. In most cases, the labels collected from the secondary emotions are discarded, since assigning a single class label to a sample is preferred from an application perspective. In this work, we take advantage of both types of annotations to improve the performance of emotion classification. We collect the labels from all the annotations available for a sample and generate primary and secondary emotion labels. A classifier is then trained using multitask learning with both primary and secondary emotions. We experimentally show that considering secondary emotion labels during the learning process leads to relative improvements of 7.9% in F1-score for an 8-class emotion classification task.

Survey of Deep Representation Learning for Speech Emotion Recognition

“…They base their experiments on three datasets for evaluations and demonstrate that the proposed MTRL model gains a concordance correlation coefficient (CCC) as high as 4.7% for within-corpus and 14.0% for cross-corpora experiments compared to STL. In [112], the authors introduce an MTRL framework for jointly learning primary and secondary emotions. They perform evaluations on the MSP-Podcast database and show that the proposed MTRL model can leverage the extra information about the secondary emotions and leads to relative improvements of 7.9% in F1-score for an 8-class emotion classification task.…”

Language-Sensitive Music Emotion Recognition Models: are We Really There Yet?

“…Since the exact position of an adjective in this space may be imprecise [21], we reduced the possibilities by defining four categories of quadrants following [22]: Q1 (positive arousal -positive valence), Q2 (positive arousal -negative valence), Q3 (negative arousal -negative valence), and Q4 (negative arousal -positive valence). By employing multi-task learning (MTL), our classifier was trained simultaneously for four-class classification (in the case of quadrants) and for binary classification (positive and negative arousal and valence) in order to improve generalization [23,24]. Our results showed feasibility of our hypothesis: intra-linguistic settings (e.g., pretraining on speech in English and fine-tuning with music in English) yield better results than cross-linguistic settings (e.g., pretraining on speech in Mandarin and fine-tuning with music in English).…”

Perceptual Borderline for Balancing Multi-Class Spontaneous Emotional Data

“…Moreover, this subset is strongly dependent of the task. For instance, a political debate on TV [16] or a podcast interview [17] cannot be expected to show the same human emotions than a human-machine scenario [10], [18], [19]. Indeed, fear is not expected in none of the mentioned corpus.…”