Adaptive Data Boosting Technique for Robust Personalized Speech Emotion in Emotionally-Imbalanced Small-Sample Environments

Abstract

Personalized emotion recognition provides an individual training model for each target user in order to mitigate the accuracy problem when using general training models collected from multiple users. Existing personalized speech emotion recognition research has a cold-start problem that requires a large amount of emotionally-balanced data samples from the target user when creating the personalized training model. Such research is difficult to apply in real environments due to the difficulty of collecting numerous target user speech data with emotionally-balanced label samples. Therefore, we propose the Robust Personalized Emotion Recognition Framework with the Adaptive Data Boosting Algorithm to solve the cold-start problem. The proposed framework incrementally provides a customized training model for the target user by reinforcing the dataset by combining the acquired target user speech with speech from other users, followed by applying SMOTE (Synthetic Minority Over-sampling Technique)-based data augmentation. The proposed method proved to be adaptive across a small number of target user datasets and emotionally-imbalanced data environments through iterative experiments using the IEMOCAP (Interactive Emotional Dyadic Motion Capture) database.

Multi-Path and Group-Loss-Based Network for Speech Emotion Recognition in Multi-Domain Datasets

“…Furthermore, collecting SER speech samples and tagging them with emotion labels is time-consuming and expensive. Thus, to overcome the limitations of volume and diversity of labeled speech samples for deep-learning SER models, studies have been performed using data augmentation [ 11 , 12 , 40 , 41 , 42 ], active learning [ 12 , 43 ] based on collected datasets, and domain adaptation [ 8 , 9 , 10 , 13 , 14 , 15 , 16 ] to adapt the existing SER datasets to the target domains.…”

“…Active-learning methods have been used to present greedy selection methods of speech samples to construct an initial SER model suitable for a target speaker based on limited samples [ 12 , 43 ]. Abdelwahab et al [ 43 ] proposed the active learning of greedy sampling to select the most informative samples to improve the performance of DNN-based SER models.…”

“…Abdelwahab et al [ 43 ] proposed the active learning of greedy sampling to select the most informative samples to improve the performance of DNN-based SER models. In a study by Bang et al [ 12 ], samples that were close to the target speaker’s samples in the embedding space were selected; the synthetic minority oversampling technique was applied to increase the number of samples of the minority class.…”

“…Many studies have effectively utilized limited emotion databases to improve the SER performance. In addition to the typical augmentation methods of speech samples [ 11 , 12 ], there exists a domain adaptation method that utilizes speech datasets already established in the unknown target domain [ 8 , 9 , 10 , 13 , 14 , 15 , 16 ]. In comparison with the results of data augmentation in a single domain, it is difficult to guarantee good performance because of the high variability of the acoustic features of the emotional speech samples in the domain [ 8 , 9 , 10 , 13 , 14 ].…”

Hardware for Recognition of Human Activities: A Review of Smart Home and AAL Related Technologies

“…Thus arose the need to develop solutions around smart home concepts using hardware and software capable of capturing residents' behavior and understanding their activities, informing them of risk situations, or taking action for their satisfaction [9]. Event recognition and emotion recognition are also part of this technology concept [10]. The smart home is considered as a technology that can help reduce the cost of living and care for the elderly and disabled population, and improve their quality of life.…”