Attention Based Fully Convolutional Network for Speech Emotion Recognition

Abstract

Speech emotion recognition is a challenging task for three main reasons: 1) human emotion is abstract, which means it is hard to distinguish; 2) in general, human emotion can only be detected in some specific moments during a long utterance; 3) speech data with emotional labeling is usually limited. In this paper, we present a novel attention based fully convolutional network for speech emotion recognition. We employ fully convolutional network as it is able to handle variable-length speech, free of the demand of segmentation to keep critical information not lost. The proposed attention mechanism can make our model be aware of which time-frequency region of speech spectrogram is more emotion-relevant. Considering limited data, the transfer learning is also adapted to improve the accuracy. Especially, it's interesting to observe obvious improvement obtained with natural scene image based pre-trained model. Validated on the publicly available IEMOCAP corpus, the proposed model outperformed the state-of-the-art methods with a weighted accuracy of 70.4% and an unweighted accuracy of 63.9% respectively.

Deep Fusion: An Attention Guided Factorized Bilinear Pooling for Audio-video Emotion Recognition

“…In [17], we proposed a novel attention based fully convolutional neural network for audio emotion recognition. The proposed attention mechanism helps the model focus on the emotion-relevant regions in speech spectrogram.…”

“…The typical CNNs, including AlexNet [23], VGGNet [24], and ResNet [25] take a fixed-size input due to the limitation of fully connected layers. Considering the loss of information caused by the fixed-size input, we proposed a fully convolutional network to handle variable-length speech in [17]. In this study, the same is used as audio encoder, which is shown in Fig.…”

“…The effectiveness of audio stream for emotion recognition has been proven in [17]. In order to verify that the attention mechanism is also effective for visual sequences, we first design a system in which only video is used, i.e., we simply remove the audio stream and the FBP block in Fig.…”

“…For the audio stream, the process in [17] is applied to extract the audio feature from raw waveform. First, a sequence of overlapping Hamming windows are applied to the speech waveform, with window shift set to 10 msec, and window size No 62.48% CNN+LMED+LSTM [32] No 61.87% set to 40 msec.…”

“…With the emerging deep learning, the state-of-the-art classifiers are always CNNs or LSTMs [14], [15], [16]. In [17], we proposed a novel attention based fully convolutional network for this task.…”