Deep convolutional recurrent neural network with attention mechanism for robust speech emotion recognition

Abstract

Deep convolutional neural networks are being actively investigated in a wide range of speech and audio processing applications including speech recognition, audio event detection and computational paralinguistics, owing to their ability to reduce factors of variations, for learning from speech. However, studies have suggested to favor a certain type of convolutional operations when building a deep convolutional neural network for speech applications although there has been promising results using different types of convolutional operations. In this work, we study four types of convolutional operations on different input features for speech emotion recognition under noisy and clean conditions in order to derive a comprehensive understanding. Since affective behavioral information has been shown to reflect temporally varying of mental state and convolutional operation are applied locally in time, all deep neural networks share a deep recurrent sub-network architecture for further temporal modeling. We present detailed quantitative module-wise performance analysis to gain insights into information flows within the proposed architectures. In particular, we demonstrate the interplay of affective information and the other irrelevant information during the progression from one module to another. Finally we show that all of our deep neural networks provide state-of-the-art performance on the eNTERFACE'05 corpus.

Speaker-Invariant Affective Representation Learning via Adversarial Training

“…The sequence of acoustic spectral features is first input to multiple 1D CNN layers. The CNN kernel filters shift along the temporal axis and include the entire spectrum information per scan, which is proven to have better performance than other kernel structure settings by [21]. CNN filters with different weights are utilized to extract different information from same input features and followed by recurrent layers to capture context and dynamic information within each speech segment.…”

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

“…The CLDNNs model is trained on the log-Mel filter bank energies [14] and on the raw waveform speech signal [15] for speech recognition, and the results showed that both CLDNN models outperform CNN and LSTM alone or combined. Similarly, in [16] and [17] CLDNN-based speech emotion recognition experiments are conducted on log-Mels and spectrograms respectively. In [18], a network architecture of convolutional recurrent neural network (CRNN) is proposed for large vocabulary speech recognition by combining the CNN and LSTM-RNN.…”

Multi-condition training for noise-robust speech emotion recognition

“…In this paper, we selected the noise signals assuming the environments in which a speech emotion recognition system is likely to be used. Some of these noise conditions are those of conventional studies (e.g., the car) [15,17]. The first three noise signals were mixed with the training and the validation sets, and the other two noise signals were mixed with the test set.…”

“…Such techniques seem to be effective for enhancing noise tolerance even in speech emotion recognition. Some studies introduced multi-condition training to speech emotion recognition [15][16][17]. For example, Heracleous et al [15] introduced multi-condition training into the i-vector and PLDA approach for recognizing the emotion of speech.…”

“…For example, Heracleous et al [15] introduced multi-condition training into the i-vector and PLDA approach for recognizing the emotion of speech. In the LLD-based approach, Eyben et al [16] and Huang et al [17] evaluated the performance of multi-condition training. Although they employed SVM as the classifier, this paper evaluates the effect of multi-condition training for DNN with LLD-based features.…”