X-Vectors Meet Emotions: A Study On Dependencies Between Emotion and Speaker Recognition

Abstract

In this work, we explore the dependencies between speaker recognition and emotion recognition. We first show that knowledge learned for speaker recognition can be reused for emotion recognition through transfer learning. Then, we show the effect of emotion on speaker recognition. For emotion recognition, we show that using a simple linear model is enough to obtain good performance on the features extracted from pre-trained models such as the x-vector model. Then, we improve emotion recognition performance by finetuning for emotion classification. We evaluated our experiments on three different types of datasets: IEMOCAP, MSP-Podcast, and Crema-D. By fine-tuning, we obtained 30.40%, 7.99%, and 8.61% absolute improvement on IEMOCAP, MSP-Podcast, and Crema-D respectively over baseline model with no pre-training. Finally, we present results on the effect of emotion on speaker verification. We observed that speaker verification performance is prone to changes in test speaker emotions. We found that trials with angry utterances performed worst in all three datasets. We hope our analysis will initiate a new line of research in the speaker recognition community.

Automatic Prediction of Speech Intelligibility Based on X-Vectors in the Context of Head and Neck Cancer

“…As mentioned before, x-vectors are DNN speaker embeddings that have seen a growing use in speaker recognition and paralinguistic tasks [16]. While i-vectors represent the total variability subspace of a channel or speaker, x-vectors aim to represent discriminative features between speakers.…”

“…Proposed by Snyder, x-vectors [13] are discriminative DNN speaker embeddings that have outperformed i-vectors in tasks such as speaker and language recognition [14,15]. Recent advances suggest that x-vectors have been successfully applied to paralinguistic tasks such as emotion recognition [16], and to the detection of diseases like Obstructive Sleep Apnea [17] and Alzheimer's [18]. Following the line of research present in [11] and [12], we investigate the reliability of using x-vector speaker embeddings as features for automatic intelligibility prediction in the context of HNC.…”

Emotion Recognition by Fusing Time Synchronous and Time Asynchronous Representations

“…In this paper, 40-dimensional (-d) FBKs with a 10 ms frame duration and 25 ms frame length are used, which is denoted FBK25. FBK features have information about the short-term spectrum but do not contain pitch information that can be important in describing emotional speech [20] and is often complementary to FBKs [21,22]. The log pitch frequency features with probability-of-voicing-weighted mean subtraction over a 1.5 second window are used along with FBKs [23].…”

The Role of Task and Acoustic Similarity in Audio Transfer Learning: Insights from the Speech Emotion Recognition Case

“…Although DNNs are already outperforming traditional approaches [21], that is not true for all tasks and data sets [22]. This has led the community to adopt transfer learning approaches, starting from feature-based [23] and recently moving to DL approaches [24,25,26]. Hence, understanding how transfer learning works could lead to the design of more powerful algorithms that unlock the full potential of DL for SER, and other low-resource audio tasks.…”