The Impact of Attention Mechanisms on Speech Emotion Recognition

Chen, Shouyan; Zhang, Mingyan; Yang, Xiaofen; Zhao, Zhijia; Zou, Tao; Sun, Xiaowei

doi:10.3390/s21227530

Cited by 18 publications

(12 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Attention mechanisms enable the model to consider the context and significance of a given input feature in relation to others in the sequence. If attention mechanisms are implemented, they improve the performance of deep learningbased speech emotion recognition systems [25]. Among the attention mechanisms mentioned earlier, multi-head attention used in [13] considers global contextual relationships among features in a parallel manner.…”

Section: Related Workmentioning

confidence: 99%

Attention-Based Multi-Learning Approach for Speech Emotion Recognition With Dilated Convolution

2022

View full text Add to dashboard Cite

The success of deep learning in speech emotion recognition has led to its application in resource-constrained devices. It has been applied in human-to-machine interaction applications like social living assistance, authentication, health monitoring and alertness systems. In order to ensure a good user experience, robust, accurate and computationally efficient deep learning models are necessary. Recurrent neural networks (RNN) like long short-term memory (LSTM), gated recurrent units (GRU) and their variants that operate sequentially are often used to learn time series sequences of the signal, analyze longterm dependencies and the contexts of the utterances in the speech signal. However, due to their sequential operation, they encounter problems in convergence and sluggish training that uses a lot of memory resources and encounters the vanishing gradient problem. In addition, they do not consider spatial cues that may exist in the speech signal. Therefore, we propose an attention-based multi-learning model (ABMD) that uses residual dilated causal convolution (RDCC) blocks and dilated convolution (DC) layers with multi-head attention. The proposed ABMD model achieves comparable performance while taking global contextualized long-term dependencies between features in a parallel manner using a large receptive field with less increase in the number of parameters compared to the number of layers and considers spatial cues among the speech features. Spectral and voice quality features extracted from the raw speech signals are used as inputs. The proposed ABMD model obtained a recognition accuracy and F1 score of 93.75% and 92.50% on the SAVEE datasets, 85.89% and 85.34% on the RAVDESS datasets and 95.93% and 95.83% on the EMODB datasets. The model's robustness in terms of the confusion ratio of the individual discrete emotions especially happiness which is often confused with emotions that belong to the same dimensional plane with it also improved when validated on the same datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Attention-Based Multi-Learning Approach for Speech Emotion Recognition With Dilated Convolution

2022

View full text Add to dashboard Cite

show abstract

“…We are motivated that the careful use of attention mechanisms in combination with other deep learning techniques allows the model to take advantage of the merits of each. Moreover, it is stated in [23] that global attention mechanisms are suitable for SER.…”

Section: Related Workmentioning

confidence: 99%

Deep Learning-Based Speech Emotion Recognition Using Multi-Level Fusion of Concurrent Features

2022

View full text Add to dashboard Cite

The detection and classification of emotional states in speech involves the analysis of audio signals and text transcriptions. There are complex relationships between the extracted features at different time intervals which ought to be analyzed to infer the emotions in speech. These relationships can be represented as spatial, temporal and semantic tendency features. In addition to emotional features that exist in each modality, the text modality consists of semantic and grammatical tendencies in the uttered sentences. Spatial and temporal features have been extracted sequentially in deep learning-based models using convolutional neural networks (CNN) followed by recurrent neural networks (RNN) which may not only be weak at the detection of the separate spatial-temporal feature representations but also the semantic tendencies in speech. In this paper, we propose a deep learning-based model named concurrent spatial-temporal and grammatical (CoSTGA) model that concurrently learns spatial, temporal and semantic representations in the local feature learning block (LFLB) which are fused as a latent vector to form an input to the global feature learning block (GFLB). We also investigate the performance of multi-level feature fusion compared to single-level fusion using the multi-level transformer encoder model (MLTED) that we also propose in this paper. The proposed CoSTGA model uses multi-level fusion first at the LFLB level where similar features (spatial or temporal) are separately extracted from a modality and secondly at the GFLB level where the spatial-temporal features are fused with the semantic tendency features. The proposed CoSTGA model uses a combination of dilated causal convolutions (DCC), bidirectional long short-term memory (BiLSTM), transformer encoders (TE), multi-head and self-attention mechanisms. Acoustic and lexical features were extracted from the interactive emotional dyadic motion capture (IEMOCAP) dataset. The proposed model achieves 75.50% and 75.82% of weighted and unweighted accuracy, 75.32% and 75.57% of recall and F1 score respectively. These results imply that concurrently learned spatial-temporal features with semantic tendencies learned in a multi-level approach improve the model's effectiveness and robustness.

show abstract

“…It takes the feature map as input to obtain a feature vector containing semantic correlation. e attention vector is obtained by (7), and the output 􏽢…”

Section: Multiscale Feature Fusion Attention Mechanismmentioning

confidence: 99%

“…Although high-level features contain rich semantic information, they cannot capture long-term relationships well. Therefore, global pooling [ 4 ], dilated convolution [ 5 ], pyramid pooling [ 6 ], and attention mechanisms [ 7 ] are used to better aggregate context. Deeplabv3+ [ 20 ] fuses features of different scales to refine the object boundaries of the segmentation results.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Semisupervised Semantic Segmentation with Mutual Correction Learning

Xiao

Dong

Zhou

et al. 2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

The semisupervised semantic segmentation method uses unlabeled data to effectively reduce the required labeled data, and the pseudo supervision performance is greatly influenced by pseudo labels. Therefore, we propose a semisupervised semantic segmentation method based on mutual correction learning, which effectively corrects the wrong convergence direction of pseudo supervision. The well-calibrated segmentation confidence maps are generated through the multiscale feature fusion attention mechanism module. More importantly, using internal knowledge, a mutual correction mechanism based on consistency regularization is proposed to correct the convergence direction of pseudo labels during cross pseudo supervision. The multiscale feature fusion attention mechanism module and mutual correction learning improve the accuracy of the entire learning process. Experiments show that the MIoU (mean intersection over union) reaches 75.32%, 77.80%, 78.95%, and 79.16% using 1/16, 1/8, 1/4, and 1/2 labeled data on PASCAL VOC 2012. The results show that the new approach achieves an advanced level.

show abstract

The Impact of Attention Mechanisms on Speech Emotion Recognition

Cited by 18 publications

References 21 publications

Attention-Based Multi-Learning Approach for Speech Emotion Recognition With Dilated Convolution

Attention-Based Multi-Learning Approach for Speech Emotion Recognition With Dilated Convolution

Deep Learning-Based Speech Emotion Recognition Using Multi-Level Fusion of Concurrent Features

Semisupervised Semantic Segmentation with Mutual Correction Learning

Contact Info

Product

Resources

About