Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Liu, Xia; Xu, Zhijing; Huang, Kan

doi:10.1155/2023/9645611

Cited by 11 publications

(4 citation statements)

References 53 publications

(58 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most companies depend on the reactions that clients have to all of their products and services. Intelligent systems make it possible to determine whether a customer is interested in a product or service based on their emotional response to a captured image or video [13] people have elaborated emotional profiles on their face, these words are universal.…”

Section: Literature Surveymentioning

confidence: 99%

CNN Based Face Emotion Recognition System for Healthcare Application

Kishore Kanna,

Panigrahi,

Sahoo

et al. 2024

EAI Endorsed Trans Perv Health Tech

View full text Add to dashboard Cite

INTRODUCTION: Because it has various benefits in areas such psychology, human-computer interaction, and marketing, the recognition of facial expressions has gained a lot of attention lately. OBJECTIVES: Convolutional neural networks (CNNs) have shown enormous potential for enhancing the accuracy of facial emotion identification systems. In this study, a CNN-based approach for recognizing facial expressions is provided. METHODS: To boost the model's generalizability, transfer learning and data augmentation procedures are applied. The recommended strategy defeated the existing state- of-the-art models when examined on multiple benchmark datasets, including the FER-2013, CK+, and JAFFE databases. RESULTS: The results suggest that the CNN-based approach is fairly excellent at properly recognizing face emotions and has a lot of potential for usage in detecting facial emotions in practical scenarios. CONCLUSION: Several diverse forms of information, including oral, textual, and visual, maybe applied to comprehend emotions. In order to increase prediction accuracy and decrease loss, this research recommended a deep CNN model for emotion prediction from facial expression.

show abstract

Section: Literature Surveymentioning

confidence: 99%

CNN Based Face Emotion Recognition System for Healthcare Application

Kishore Kanna,

Panigrahi,

Sahoo

et al. 2024

EAI Endorsed Trans Perv Health Tech

View full text Add to dashboard Cite

show abstract

“…The article [41] three corpus containing e-commerce data belonging to different languages such as Turkish, Arabic, and English were created and the performances of deep learning and machine learning methods were examined comparatively by performing sentiment analysis on them. The study [42] offered a unique multimodal emotion identification framework that names multimodal emotion identification based on cascaded multichannel and hierarchical fusion (CMC-HF). This framework makes use of visual, voice, and text data as multimodal inputs concurrently.…”

Section: Literature Reviewmentioning

confidence: 99%

TER-CA-WGNN: Trimodel Emotion Recognition Using Cumulative Attribute-Weighted Graph Neural Network

Al-Saadawi,

Das

2024

Applied Sciences

View full text Add to dashboard Cite

Affective computing is a multidisciplinary field encompassing artificial intelligence, natural language processing, linguistics, computer science, and social sciences. This field aims to deepen our comprehension and capabilities by deploying inventive algorithms. This article presents a groundbreaking approach, the Cumulative Attribute-Weighted Graph Neural Network, which is innovatively designed to integrate trimodal textual, audio, and visual data from the two multimodal datasets. This method exemplifies its effectiveness in performing comprehensive multimodal sentiment analysis. Our methodology employs vocal inputs to generate speaker embeddings trimodal analysis. Using a weighted graph structure, our model facilitates the efficient integration of these diverse modalities. This approach underscores the interrelated aspects of various emotional indicators. The paper’s significant contribution is underscored by its experimental results. Our novel algorithm achieved impressive performance metrics on the CMU-MOSI dataset, with an accuracy of 94% and precision, recall, and F1-scores above 92% for Negative, Neutral, and Positive emotion categories. Similarly, on the IEMOCAP dataset, the algorithm demonstrated its robustness with an overall accuracy of 93%, where exceptionally high precision and recall were noted in the Neutral and Positive categories. These results mark a notable advancement over existing state-of-the-art models, illustrating the potential of our approach in enhancing Sentiment Recognition through the synergistic use of trimodal data. This study’s comprehensive analysis and significant results demonstrate the proposed algorithm’s effectiveness in nuanced emotional state recognition and pave the way for future advancements in affective computing, emphasizing the value of integrating multimodal data for improved accuracy and robustness.

show abstract

“…In the case of multimodal emotion recognition from facial and speech features, deep neural networks [ 13 , 14 , 15 ] have been employed to extract relevant features from each modality. Networks learn to extract features relevant to emotion classification.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features

Dilnoza

Abdusalomov

Kutlimuratov

et al. 2023

Sensors

View full text Add to dashboard Cite

Methods for detecting emotions that employ many modalities at the same time have been found to be more accurate and resilient than those that rely on a single sense. This is due to the fact that sentiments may be conveyed in a wide range of modalities, each of which offers a different and complementary window into the thoughts and emotions of the speaker. In this way, a more complete picture of a person’s emotional state may emerge through the fusion and analysis of data from several modalities. The research suggests a new attention-based approach to multimodal emotion recognition. This technique integrates facial and speech features that have been extracted by independent encoders in order to pick the aspects that are the most informative. It increases the system’s accuracy by processing speech and facial features of various sizes and focuses on the most useful bits of input. A more comprehensive representation of facial expressions is extracted by the use of both low- and high-level facial features. These modalities are combined using a fusion network to create a multimodal feature vector which is then fed to a classification layer for emotion recognition. The developed system is evaluated on two datasets, IEMOCAP and CMU-MOSEI, and shows superior performance compared to existing models, achieving a weighted accuracy WA of 74.6% and an F1 score of 66.1% on the IEMOCAP dataset and a WA of 80.7% and F1 score of 73.7% on the CMU-MOSEI dataset.

show abstract

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Cited by 11 publications

References 53 publications

CNN Based Face Emotion Recognition System for Healthcare Application

CNN Based Face Emotion Recognition System for Healthcare Application

TER-CA-WGNN: Trimodel Emotion Recognition Using Cumulative Attribute-Weighted Graph Neural Network

Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features

Contact Info

Product

Resources

About