Fusion of deep learning features with mixture of brain emotional learning for audio-visual emotion recognition

Farhoudi, Zeinab; Setayeshi, Saeed

doi:10.1016/j.specom.2020.12.001

Cited by 36 publications

(19 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For multimodal fusion, one of the major challenges is how to effectively integrate data from different sources and design moderate architecture to complete representation learning. According to the fusion level or location, fusion concept can be divided into three types: data-level, featurelevel and decision-level [24]. As the development of the machine learning, feature-level and decision-level fusion (or called early and late fusion) have spawned a variety of studies.…”

Section: Related Workmentioning

confidence: 99%

A novel dual-modal emotion recognition algorithm with fusing hybrid features of audio signal and speech context

et al. 2022

Complex Intell. Syst.

View full text Add to dashboard Cite

With regard to human–machine interaction, accurate emotion recognition is a challenging problem. In this paper, efforts were taken to explore the possibility to complete the feature abstraction and fusion by the homogeneous network component, and propose a dual-modal emotion recognition framework that is composed of a parallel convolution (Pconv) module and attention-based bidirectional long short-term memory (BLSTM) module. The Pconv module employs parallel methods to extract multidimensional social features and provides more effective representation capacity. Attention-based BLSTM module is utilized to strengthen key information extraction and maintain the relevance between information. Experiments conducted on the CH-SIMS dataset indicate that the recognition accuracy reaches 74.70% on audio data and 77.13% on text, while the accuracy of the dual-modal fusion model reaches 90.02%. Through experiments it proves the feasibility to process heterogeneous information within homogeneous network component, and demonstrates that attention-based BLSTM module would achieve best coordination with the feature fusion realized by Pconv module. This can give great flexibility for the modality expansion and architecture design.

show abstract

Section: Related Workmentioning

confidence: 99%

A novel dual-modal emotion recognition algorithm with fusing hybrid features of audio signal and speech context

et al. 2022

Complex Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…The OFC evaluates the response of the amygdala based on the input received from the Sensory cortex, which leads to the prevention of improper learning connections. In a nutshell, the Sensory cortex integrates the features extracted from different unimodal inputs and the amygdala and OFC works to form a decision after interacting with the memory [17,18]. BEL models require rewards extracted from input data and are derived from monotonic reinforcement learning.…”

Section: Brain Emotion Learning Modelsmentioning

confidence: 99%

Review on Multimodal Fusion Techniques for Human Emotion Recognition

Karani¹,

Desai²

2022

IJACSA

View full text Add to dashboard Cite

Emotions play an essential role in human life for planning and decision making. Emotion identification and recognition is a widely explored field in the area of artificial intelligence and affective computing as a means of empathizing with humans and thereby improving human machine interaction. Though audio visual cues are vital for recognizing human emotions, they are sometimes insufficient in identifying emotions of people who are good at hiding emotions or people suffering from Alexithymia. Considering other dimensions like Electroencephalogram (EEG) or text, along with audio visual cues can aid in improving the results in such situations. Taking advantage of the complementarity of multiple modalities normally helps capture emotions more accurately compared to single modality. However, to achieve precise and accurate results, correct fusion of these multimodal signals is solicited. This study provides a detailed review of different multimodal fusion techniques that can be used for emotion recognition. This paper proposes in-depth study of feature-level fusion, decision-level fusion and hybrid fusion techniques for identifying human emotions based on multimodal inputs and compare the results. The study concentrates on three different modalities i.e., facial images, audio and text for experimentation; at least one of which differs in temporal characteristics. The result suggests that hybrid fusion works best in combining multiple modalities which differ in time synchronicity.

show abstract

“…When signals propagate through the network, a conditional layer learns which expert networks to activate, and so that the various combinations of experts are flexible under different circumstances. It was proven to outperform popular fusion strategies in dynamic emotion prediction using visual-audio [49], [50]. For developing DMoE model, we apply fully-connected layers for both expert and gating networks.…”

Section: Multimodal Fusion Modulesmentioning

confidence: 99%

Deep Graph Fusion Based Multimodal Evoked Expressions From Large-Scale Videos

Yang

Kim

et al. 2021

IEEE Access

View full text Add to dashboard Cite

In the case of emotion recognition for wild input signals with large variances, multiple sources of noise can challenge the machine's ability to learn approximate ground truth. There are copious studies on recognizing characters' affective expressions directly through face, speech, and text. However, there are few researches on the prediction of a characters' emotions from their watched contents. Therefore, in this paper, we propose a hybrid fusion model, called deep graph fusion, to leverage the combination of visual-audio representations for predicting viewers' evoked expressions from videos. The proposed system includes four steps. First, we extract features for visual and audio modalities for each 30-second segment using CNN-based pre-trained models to learn their salient representations. Then, we reconstruct these features into graph outlines, and conduct node embedding using graph convolutional networks. In the third steps, we propose various fusion modules to combine the graph representations from visual and audio branches. Finally, the fused features are applied to Sigmoid activation in order to estimate the evoked scores for all emotional classes. Moreover, in order to enhance the overall performance, we propose a semantic embedding loss to learn the semantic meaning of textual emotions. We evaluate the proposed method using the Evoked Expression from Videos (EEV) database on both the validation and test sets. The experimental results demonstrate that the proposed algorithm outperforms all baseline models.

show abstract

Fusion of deep learning features with mixture of brain emotional learning for audio-visual emotion recognition

Cited by 36 publications

References 25 publications

A novel dual-modal emotion recognition algorithm with fusing hybrid features of audio signal and speech context

A novel dual-modal emotion recognition algorithm with fusing hybrid features of audio signal and speech context

Review on Multimodal Fusion Techniques for Human Emotion Recognition

Deep Graph Fusion Based Multimodal Evoked Expressions From Large-Scale Videos

Contact Info

Product

Resources

About