Generative Data Augmentation Guided by Triplet Loss for Speech Emotion Recognition

Abstract

Speech Emotion Recognition (SER) is crucial for humancomputer interaction but still remains a challenging problem because of two major obstacles: data scarcity and imbalance. Many datasets for SER are substantially imbalanced, where data utterances of one class (most often Neutral) are much more frequent than those of other classes. Furthermore, only a few data resources are available for many existing spoken languages. To address these problems, we exploit a GAN-based augmentation model guided by a triplet network, to improve SER performance given imbalanced and insufficient training data. We conduct experiments and demonstrate: 1) With a highly imbalanced dataset, our augmentation strategy significantly improves the SER performance (+8% recall score compared with the baseline). 2) Moreover, in a cross-lingual benchmark, where we train a model with enough source language utterances but very few target language utterances (around 50 in our experiments), our augmentation strategy brings benefits for the SER performance of all three target languages.

Towards Improving Speech Emotion Recognition Using Synthetic Data Augmentation from Emotion Conversion

“…Generating spectrograms or raw waveforms provides more flexibility by allowing us to train models directly on the raw data. Chatziagapi et al [4] and Wang et al [5] proposed generating mel spectrograms using GANs to tackle data imbalance by augmenting the minority classes. Similarly Eskimez et al [16] used an improved version of GANs with higher generation quality to apply SER data augmentation using spectrograms.…”

“…Synthetic data is artificially generated data, which can be used to replace or augment real data in training deep learning models. Such approach has multiple advantages in terms of data privacy and security [3], balancing skewed datasets [4,5], as well as overcoming the lack of large datasets, as the case with SER [6]. The quality and realism of synthetic data is critical for its effectiveness in deep learning applications.…”

Multimodal and Multitask Learning with Additive Angular Penalty Focus Loss for Speech Emotion Recognition

“…Data augmentation is an efective method to solve this problem [16]. For example, generative adaptive networks (GANs) and its variants are often applied to generate new samples [17][18][19]. Alternatively, a larger data can be directly constructed from existing data with hand-crafted features [20].…”