Speech Expression Multimodal Emotion Recognition Based on Deep Belief Network

Liu, Dong; Chen, Longxi; Wang, Zhiyong; Diao, Guangqiang

doi:10.1007/s10723-021-09564-0

Cited by 25 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Tensor Fusion Network‐based SER [33] learnt both intra‐ and inter‐modality dynamics. A convolutional deep belief network‐based SER [34] learnt the multimodal feature representations linked to the expressions. The single‐plain CNN model is weak at classifying the speaker's emotional state with the required accuracy level because it drops basic sequential information during the convolutional operations.…”

Section: Related Literaturementioning

confidence: 99%

DeepCNN: Spectro‐temporal feature representation for speech emotion recognition

Saleem

Gao

Irfan

et al. 2023

CAAI Trans on Intel Tech

View full text Add to dashboard Cite

Speech emotion recognition (SER) is an important research problem in human-computer interaction systems. The representation and extraction of features are significant challenges in SER systems. Despite the promising results of recent studies, they generally do not leverage progressive fusion techniques for effective feature representation and increasing receptive fields. To mitigate this problem, this article proposes DeepCNN, which is a fusion of spectral and temporal features of emotional speech by parallelising convolutional neural networks (CNNs) and a convolution layer-based transformer. Two parallel CNNs are applied to extract the spectral features (2D-CNN) and temporal features (1D-CNN) representations. A 2D-convolution layer-based transformer module extracts spectro-temporal features and concatenates them with features from parallel CNNs. The learnt low-level concatenated features are then applied to a deep framework of convolutional blocks, which retrieves high-level feature representation and subsequently categorises the emotional states using an attention gated recurrent unit and classification layer. This fusion technique results in a deeper hierarchical feature representation at a lower computational cost while simultaneously expanding the filter depth and reducing the feature map. The Berlin Database of Emotional Speech (EMO-BD) and Interactive Emotional Dyadic Motion Capture (IEMOCAP) datasets are used in experiments to recognise distinct speech emotions. With efficient spectral and temporal feature representation, the proposed SER model achieves 94.2% accuracy for different emotions on the EMO-BD and 81.1% accuracy on the IEMOCAP dataset respectively. The proposed SER system, DeepCNN, outperforms the baseline SER systems in terms of emotion recognition accuracy on the EMO-BD and IEMOCAP datasets.

show abstract

Section: Related Literaturementioning

confidence: 99%

DeepCNN: Spectro‐temporal feature representation for speech emotion recognition

Saleem

Gao

Irfan

et al. 2023

CAAI Trans on Intel Tech

View full text Add to dashboard Cite

show abstract

“…With the improvement of chip computing processing power and deep learning performance, many novel emotion recognition methods have emerged in recent years. Some mainstream neural network models have achieved good results in emotion recognition, such as CNN [4], LSTM [8], [20], DBN [23], and GCN [24]. These deep learning methods have gradually replaced traditional feature extraction methods as the primary research methods for emotion recognition.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Pan

Fang

Zhang

et al. 2024

IEEE Open J. Eng. Med. Biol.

View full text Add to dashboard Cite

As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

show abstract

“…Multiple hidden layers are deployed in DL methods for solving complex pattern recognition problems. DL is widely used in healthcare applications such as ulcer detection [6,7], predicting the risk of heart failure [8], discovering the gait patterns which reveal the neurodegenerative diseases [9], Abnormal Behavior Detection [10], Dim Target Detection [11], Emotion Recognition [12] and classifying ECG signal database [13].…”

Section: Introductionmentioning

confidence: 99%

A Deep Learning and Handcrafted Based Computationally Intelligent Technique for Effective COVID-19 Detection from X-ray/CT-scan Imaging

2022

View full text Add to dashboard Cite

The world has witnessed dramatic changes because of the advent of COVID19 in the last few days of 2019. During the last more than two years, COVID-19 has badly affected the world in diverse ways. It has not only affected human health and mortality rate but also the economic condition on a global scale. There is an urgent need today to cope with this pandemic and its diverse effects. Medical imaging has revolutionized the treatment of various diseases during the last four decades. Automated detection and classification systems have proven to be of great assistance to the doctors and scientific community for the treatment of various diseases. In this paper, a novel framework for an efficient COVID-19 classification system is proposed which uses the hybrid feature extraction approach. After preprocessing image data, two types of features i.e., deep learning and handcrafted, are extracted. For Deep learning features, two pre-trained models namely ResNet101 and DenseNet201 are used. Handcrafted features are extracted using Weber Local Descriptor (WLD). The Excitation component of WLD is utilized and features are reduced using DCT. Features are extracted from both models, handcrafted features are fused, and significant features are selected using entropy. Experiments have proven the effectiveness of the proposed model. A comprehensive set of experiments have been performed and results are compared with the existing well-known methods. The proposed technique has performed better in terms of accuracy and time.

show abstract

Speech Expression Multimodal Emotion Recognition Based on Deep Belief Network

Cited by 25 publications

References 25 publications

DeepCNN: Spectro‐temporal feature representation for speech emotion recognition

DeepCNN: Spectro‐temporal feature representation for speech emotion recognition

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

A Deep Learning and Handcrafted Based Computationally Intelligent Technique for Effective COVID-19 Detection from X-ray/CT-scan Imaging

Contact Info

Product

Resources

About