Hierarchical Network Based on the Fusion of Static and Dynamic Features for Speech Emotion Recognition

Abstract

Many studies on automatic speech emotion recognition (SER) have been devoted to extracting meaningful emotional features for generating emotion-relevant representations. However, they generally ignore the complementary learning of static and dynamic features, leading to limited performances. In this paper, we propose a novel hierarchical network called HNSD that can efficiently integrate the static and dynamic features for SER. Specifically, the proposed HNSD framework consists of three different modules. To capture the discriminative features, an effective encoding module is firstly designed to simultaneously encode both static and dynamic features. By taking the obtained features as inputs, the Gated Multi-features Unit (GMU) is conducted to explicitly determine the emotional intermediate representations for framelevel features fusion, instead of directly fusing these acoustic features. In this way, the learned static and dynamic features can jointly and comprehensively generate the unified feature representations. Benefiting from a well-designed attention mechanism, the last classification module is applied to predict the emotional states at the utterance level. Extensive experiments on the IEMOCAP benchmark dataset demonstrate the superiority of our method in comparison with state-of-the-art baselines.

Speech Emotion Recognition with Co-Attention based Multi-level Acoustic Information

“…Performance comparison with 5-fold leaveone-session-out [12,7,13] and 10-fold leave-one-speaker-out [14,15,16,17,18,5] cross-validation strategy on IEMOCAP.…”

“…Model WA UA CNN-ELM+STC attention [12] 61.32 60.43 Audio 25 [7] 60.64±1.96 61.32±2.26 IS09 -classification [13] 68.1 63.8 Ours 69.80 71.05 RNN(prop. )-ELM [14] 62.85 63.89 3D ACRNN [15] -64.74±5.44 BLSTM-CTC-CA [16] 69.0 67.0 CNN GRU-SeqCap [17] 72.73 59.71 CNN TF Att.pooling [18] 71.75 68.06 HNSD [5] 70.5 72.5 Ours 71.64 72.70…”

“…IEMOCAP is a widely used emotion recognition dataset, recorded from ten different actors with audio, video, transcriptions and motion-capture information. Following others' work [12,7,5], we merge "happy" and "excited" into the category of "happy" and we consider the 5531 acoustic utterances from 4 emotions, angry, sad, happy and neutral.…”

“…The acoustic information is mined using a series of CNNs with different kernel sizes in [3]. Some methods propose to introduce a combination of networks to extract acoustic information, e.g., [5] combine LSTM and Gated Multi-features Unit (GMU) to extract both static and dynamic speech signals. In [6], Gao et al propose a domain-adversarial auto-encoder to extract discriminative representations with pre-trained spectrogram information.…”

“…Different types of attention mechanisms have been proposed for processing the extracted features, like the commonly used self-attention [5,7] and cross-modal attention [8]. For the models with more complex input combinations, new attention mechanisms are introduced.…”