Multi-Path and Group-Loss-Based Network for Speech Emotion Recognition in Multi-Domain Datasets

Abstract

Speech emotion recognition (SER) is a natural method of recognizing individual emotions in everyday life. To distribute SER models to real-world applications, some key challenges must be overcome, such as the lack of datasets tagged with emotion labels and the weak generalization of the SER model for an unseen target domain. This study proposes a multi-path and group-loss-based network (MPGLN) for SER to support multi-domain adaptation. The proposed model includes a bidirectional long short-term memory-based temporal feature generator and a transferred feature extractor from the pre-trained VGG-like audio classification model (VGGish), and it learns simultaneously based on multiple losses according to the association of emotion labels in the discrete and dimensional models. For the evaluation of the MPGLN SER as applied to multi-cultural domain datasets, the Korean Emotional Speech Database (KESD), including KESDy18 and KESDy19, is constructed, and the English-speaking Interactive Emotional Dyadic Motion Capture database (IEMOCAP) is used. The evaluation of multi-domain adaptation and domain generalization showed 3.7% and 3.5% improvements, respectively, of the F1 score when comparing the performance of MPGLN SER with a baseline SER model that uses a temporal feature generator. We show that the MPGLN SER efficiently supports multi-domain adaptation and reinforces model generalization.

Arabic Speech Emotion Recognition Employing Wav2vec2.0 and HuBERT Based on BAVED Dataset

“…Despite the enormous success contributions in emotion recognition in English datasets, there is still gab in Arabic dataset and emotion recognition systems utilizes these Arabic datasets. Some Arabic speeches emotion datasets have been proposed in the literature, see [1]- [3], [5], [19]. Each dataset has a different set of classes or labels, for example, the Arabic audio acted dataset proposed in [20] has five labels (Happiness, Sadness, Neutral, Anger, Fear), and the dataset proposed in [15] has three classes (Happy, Surprised, and Angry), while the dataset proposed in [19] has labels (Happy, Sad, Neutral, Angry, Surprise, Disgust).…”

“…Despite the enormous success contributions in emotion recognition in English datasets, there is still a gab in Arabic dataset and emotion recognition systems utilizes these Arabic datasets. Various Arabic speeches emotion datasets have been proposed in the literature, whether audio or visual, see [1]- [4].…”

Special Issue “Emotion Intelligence Based on Smart Sensing”

“…This issue also includes a speech-emotion-recognition study [ 6 ] that proposed a multi-path and group-loss-based network (MPGLN) for emotion recognition to support multi-domain adaptation. The authors proposed a model that includes a bidirectional long short-term memory-based temporal feature generator and a transferred feature extractor from the pre-trained VGG-like audio classification model (VGGish).…”