LSTM Based Cross-corpus and Cross-task Acoustic Emotion Recognition

Abstract

Acoustic emotion recognition is a popular and central research direction in paralinguistic analysis, due its relation to a wide range of affective states/traits and manifold applications. Developing highly generalizable models still remains as a challenge for researchers and engineers, because of multitude of nuisance factors. To assert generalization, deployed models need to handle spontaneous speech recorded under different acoustic conditions compared to the training set. This requires that the models are tested for cross-corpus robustness. In this work, we first investigate the suitability of Long-Short-Term-Memory (LSTM) models trained with time-and space-continuously annotated affective primitives for cross-corpus acoustic emotion recognition. We next employ an effective approach to use the frame level valence and arousal predictions of LSTM models for utterance level affect classification and apply this approach on the ComParE 2018 challenge corpora. The proposed method alone gives motivating results both on development and test set of the Self-Assessed Affect Sub-Challenge. On the development set, the cross-corpus prediction based method gives a boost to performance when fused with top components of the baseline system. Results indicate the suitability of the proposed method for both time-continuous and utterance level cross-corpus acoustic emotion recognition tasks.

История СПБ ФИЦ РАН: 45 Лет Научной Деятельности

“…[58,59]. Создана система кросс-корпусного распознавания естественных эмоций в речи, основанная на рекуррентных нейронных сетях с долгой кратковременной памятью (LSTM), включающая предобработку признаков, доменную адаптации, обучение и предсказание значений эмоциональных дескрипторов активации и валентности, отличающаяся от аналогов интегральным использованием нескольких корпусов эмоциональной речи для обучения системы на посегментной разметке и ее применения для классификации целых высказываний [60,61]. Предложен метод извлечения геометрических визуальных признаков для описания конфигурации губ на основе 24 пар ключевых точек на компьютерных изображениях губ и рта диктора, что позволяет максимизировать точность отслеживания движений губ дикторов, отличающийся использованием видеозаписей непрерывной русской речи, полученных при помощи высокоскоростной камеры, обеспечивающий повышение точности и робастности аудиовизуального распознавания речи и чтения речи по губам говорящего в реальных условиях функционирования при наличии сильных акустических шумов [62].…”

Speech Emotion Recognition Based on Deep Residual Shrinkage Network

“…hh ∈ R h×h are weight matrix. Then the calculated → h and ← h are connected to obtain the hidden state h t ∈ R n×2h at current time, and the output layer o t ∈ R n×q is calculated by Equation (16).…”

“…In recent years, deep learning has emerged as a prominent alternative to the traditional models, showcasing superior performance across various fields, including the realm of speech emotion recognition. Recent applications of DNN, RNN, CNN, LSTM, and other network models have reaped fruitful outputs in voice emotion recognition [16,17].…”

Deep Learning for Audio Visual Emotion Recognition

“…With the recent development of DL algorithms in behavioral signal processing and affective computing, the DL-based emotion recognition algorithms have received significant attention. Notable DL-based emotion recognition approaches include Long-Short Time Memory architectures [12] [13] [14]), deep neural network (DNN) [15], convolutional neural network (CNN) [16] [17] [18], and bidirectional long short-term memory (BLSTM) [19]. Among the DL-based models, CNNs have been shown to be effective in detecting emotions, due to its capability in characterizing local temporal-spectral structures of speech and audio signals, as well as its generalisation ability and recognition accuracy.…”