Fusion-ConvBERT: Parallel Convolution and BERT Fusion for Speech Emotion Recognition

Lee, Sang‐Hyun; Han, David K.; Ko, Hanseok

doi:10.3390/s20226688

Cited by 27 publications

(16 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A recurrent neural network (RNN) can be used to infer associations from 3D spectrogram data across different timesteps and frequencies [ 42 ]. “Fusion-ConvBERT” is a parallel fusion model proposed by Lee et al [ 43 ] that comprised bidirectional encoder representations derived from transformers and convolutional neural networks together. Zhang et al [ 44 ] built a deep convolution neural network (DCNN) and a bidirectional long short-term memory with attention (BLSTMwA) model (DCNN-BLSTMwA) that can be utilized as a pretrained model for subsequent emotion recognition tasks.…”

Section: Literature Reviewmentioning

confidence: 99%

Human-Computer Interaction for Recognizing Speech Emotions Using Multilayer Perceptron Classifier

Alnuaim

Zakariah

Shukla

et al. 2022

Journal of Healthcare Engineering

View full text Add to dashboard Cite

Human-computer interaction (HCI) has seen a paradigm shift from textual or display-based control toward more intuitive control modalities such as voice, gesture, and mimicry. Particularly, speech has a great deal of information, conveying information about the speaker’s inner condition and his/her aim and desire. While word analysis enables the speaker’s request to be understood, other speech features disclose the speaker’s mood, purpose, and motive. As a result, emotion recognition from speech has become critical in current human-computer interaction systems. Moreover, the findings of the several professions involved in emotion recognition are difficult to combine. Many sound analysis methods have been developed in the past. However, it was not possible to provide an emotional analysis of people in a live speech. Today, the development of artificial intelligence and the high performance of deep learning methods bring studies on live data to the fore. This study aims to detect emotions in the human voice using artificial intelligence methods. One of the most important requirements of artificial intelligence works is data. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) open-source dataset was used in the study. The RAVDESS dataset contains more than 2000 data recorded as speeches and songs by 24 actors. Data were collected for eight different moods from the actors. It was aimed at detecting eight different emotion classes, including neutral, calm, happy, sad, angry, fearful, disgusted, and surprised moods. The multilayer perceptron (MLP) classifier, a widely used supervised learning algorithm, was preferred for classification. The proposed model’s performance was compared with that of similar studies, and the results were evaluated. An overall accuracy of 81% was obtained for classifying eight different emotions by using the proposed model on the RAVDESS dataset.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Human-Computer Interaction for Recognizing Speech Emotions Using Multilayer Perceptron Classifier

Alnuaim

Zakariah

Shukla

et al. 2022

Journal of Healthcare Engineering

View full text Add to dashboard Cite

show abstract

“…They suggested a genetic learning-based collaborative decision-making model, which was compared to concatenated equal weighted choice fusion, BPN learning-based weighted decision fusion, and feature fusion methods. The audio spectrum features are obtained from BERT and CNN and are combined in parallel to form a multimodal [32].…”

Section: Related Studies and Motivationsmentioning

confidence: 99%

Human Emotion Recognition by Integrating Facial and Speech Features: An Implementation of Multimodal Framework using CNN

Srinivas¹,

Mishra²

2022

IJACSA

View full text Add to dashboard Cite

This Emotion recognition plays a prominent role in today's intelligent system applications. Human computer interface, health care, law, and entertainment are a few of the applications where emotion recognition is used. Humans convey their emotions in the form of text, voice, and facial expressions, thus developing a multimodal emotional recognition system playing a crucial role in human-computer or intelligent system communication. The majority of established emotional recognition algorithms only identify emotions in unique data, such as text, audio, or image data. A multimodal system uses information from a variety of sources and fuses the information by using fusion techniques and categories to improve recognition accuracy. In this paper, a multimodal system to recognise emotions was presented that fuses the features from information obtained from heterogenous modalities like audio and video. For audio feature extraction energy, zero crossing rate and Mel-Frequency Cepstral Coefficients (MFCC) techniques are considered. Of these, MFCC produced promising results. For video feature extraction, first the videos are converted to frames and stored in a linear scale space by using a spatial temporal Gaussian Kernel. The features from the images are further extracted by applying a Gaussian weighted function to the second momentum matrix of linear scale space data. The Marginal Fisher Analysis (MFA) fusion method is used to fuse both the audio and video features, and the resulted features are given to the FERCNN model for evaluation. For experimentation, the RAVDESS and CREMAD datasets, which contain audio and video data, are used. Accuracy levels of 95.56, 96.28, and 95.07 on the RAVDESS dataset and accuracies of 80.50, 97.88, and 69.66 on the CREMAD dataset in audio, video, and multimodal modalities are achieved, whose performance is better than the existing multimodal systems.

show abstract

“…Tsai et al [41] proposed learning interactions between the modalities by designing an attention based cross-modal architecture using multimodal transformers. Also, recently, transfer learning techniques that use pre-trained networks to extract features [26], [62]- [64] have advanced significantly. BERT, a Transformer based model, has shown performance improvement by fine-tuning from pre-trained weights for a specific downstream task [65].…”

Section: Related Workmentioning

confidence: 99%

“…To improve performance, there have been some efforts of fusing different features extracted from a unimodal source. Lee et al [26] proposed a method that combines two features of CNN and BERT in parallel from an audio spectrogram. Xu et al [27] proposed "Hierarchical Grained and Feature Model (HGFM)" by fusion of handcrafted features and Gated Recurrent Unit (GRU) network extracted features.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification

Lee

Han

2021

IEEE Access

Self Cite

View full text Add to dashboard Cite

Human communication includes rich emotional content, thus the development of multimodal emotion recognition plays an important role in communication between humans and computers. Because of the complex emotional characteristics of a speaker, emotional recognition remains a challenge, particularly in capturing emotional cues across a variety of modalities, such as speech, facial expressions, and language. Audio and visual cues are particularly vital for a human observer in understanding emotions. However, most previous work on emotion recognition has been based solely on linguistic information, which can overlook various forms of nonverbal information. In this paper, we present a new multimodal emotion recognition approach that improves the BERT model for emotion recognition by combining it with heterogeneous features based on language, audio, and visual modalities. Specifically, we improve the BERT model due to the heterogeneous features of the audio and visual modalities. We introduce the Self-Multi-Attention Fusion module, Multi-Attention fusion module, and Video Fusion module, which are attention based multimodal fusion mechanisms using the recently proposed transformer architecture. We explore the optimal ways to combine fine-grained representations of audio and visual features into a common embedding while combining a pre-trained BERT model with modalities for fine-tuning. In our experiment, we evaluate the commonly used CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets for multimodal sentiment analysis. Ablation analysis indicates that the audio and visual components make a significant contribution to the recognition results, suggesting that these modalities contain highly complementary information for sentiment analysis based on video input. Our method shows that we achieve state-of-the-art performance on the CMU-MOSI, CMU-MOSEI, and IEMOCAP dataset.

show abstract

Fusion-ConvBERT: Parallel Convolution and BERT Fusion for Speech Emotion Recognition

Cited by 27 publications

References 51 publications

Human-Computer Interaction for Recognizing Speech Emotions Using Multilayer Perceptron Classifier

Human-Computer Interaction for Recognizing Speech Emotions Using Multilayer Perceptron Classifier

Human Emotion Recognition by Integrating Facial and Speech Features: An Implementation of Multimodal Framework using CNN

Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification

Contact Info

Product

Resources

About