Bi-modality Fusion for Emotion Recognition in the Wild

Li, Sunan; Zheng, Wenming; Zong, Yuan; Lu, Cheng; Tang, Chuangao; Jiang, Xingxun; Liu, Jiateng; Xia, Wanchuang

doi:10.1145/3340555.3355719

Cited by 35 publications

(20 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Typically, low and high-level feature extractors are combined in a base network architecture. We choose VGG16 [22] as the base network because this network is still used as the base of many recent network for the FER task [31], [32], [33]. From the base network, VGG16 [22], we separated into two parts for two levels of input.…”

Section: Pyramid With Super-resolution (Psr) Networkmentioning

confidence: 99%

Pyramid With Super Resolution for In-the-Wild Facial Expression Recognition

et al. 2020

View full text Add to dashboard Cite

Facial Expression Recognition (FER) is a challenging task that improves natural humancomputer interaction. This paper focuses on automatic FER on a single in-the-wild (ITW) image. ITW images suffer real problems of pose, direction, and input resolution. In this study, we propose a pyramid with super-resolution (PSR) network architecture to solve the ITW FER task. We also introduce a prior distribution label smoothing (PDLS) loss function that applies the additional prior knowledge of the confusion about each expression in the FER task. Experiments on the three most popular ITW FER datasets showed that our approach outperforms all the state-of-the-art methods.

show abstract

Section: Pyramid With Super-resolution (Psr) Networkmentioning

confidence: 99%

Pyramid With Super Resolution for In-the-Wild Facial Expression Recognition

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The accuracy measurements of our proposed methods and related methods on the AFEW validation set are shown in Table 6. Our spatiotemporal method outperforms other recently reported methods using the same approach, by around 0.14% compared with Li et al [63]. Recently, Kumar et al [66] used multi-level attention with an unsupervised approach by iterative training between student and teacher models.…”

Section: Discussion and Comparison With Related Workmentioning

confidence: 54%

Context-Aware Emotion Recognition in the Wild Using Spatio-Temporal and Temporal-Pyramid Models

Kim

Yang

et al. 2021

Sensors

View full text Add to dashboard Cite

Emotion recognition plays an important role in human–computer interactions. Recent studies have focused on video emotion recognition in the wild and have run into difficulties related to occlusion, illumination, complex behavior over time, and auditory cues. State-of-the-art methods use multiple modalities, such as frame-level, spatiotemporal, and audio approaches. However, such methods have difficulties in exploiting long-term dependencies in temporal information, capturing contextual information, and integrating multi-modal information. In this paper, we introduce a multi-modal flexible system for video-based emotion recognition in the wild. Our system tracks and votes on significant faces corresponding to persons of interest in a video to classify seven basic emotions. The key contribution of this study is that it proposes the use of face feature extraction with context-aware and statistical information for emotion recognition. We also build two model architectures to effectively exploit long-term dependencies in temporal information with a temporal-pyramid model and a spatiotemporal model with “Conv2D+LSTM+3DCNN+Classify” architecture. Finally, we propose the best selection ensemble to improve the accuracy of multi-modal fusion. The best selection ensemble selects the best combination from spatiotemporal and temporal-pyramid models to achieve the best accuracy for classifying the seven basic emotions. In our experiment, we take benchmark measurement on the AFEW dataset with high accuracy.

show abstract

“…The bias of nth hidden unit is represented as b n . The activation probability of mth visible unit having known hidden vector h(h 1 , … , h n , .., h k ) is computed using Equation (12),…”

Section: Ta B L E 2 Steps Applied By Contrastive Divergence Algorithmmentioning

confidence: 99%

“…Moreover, the advantages of noncontact is found in video recordings, therefore it is applied in worldwide applications. 12 Still now, number of techniques are made available for recognizing the distinct expressions from the human face. Some of the common techniques basically applied for this purpose are convolutional neural network (CNN) and deep belief network (DBN).…”

Section: Introductionmentioning

confidence: 99%

Modified deep belief network based human emotion recognition with multiscale features from video sequences

Sreenivas¹,

Namdeo²,

Kumar

2021

Softw Pract Exp

View full text Add to dashboard Cite

Emotion recognition from human faces are recently considered as growing topic for the applications in HCI (human-computer interaction) field. Therefore, a new framework is introduced in this method for emotion recognition from video. Human faces may carry huge features which increase the complexity of recognizing the emotions from the give video. Therefore, to minimize such defect, the wrapper based feature selection technique is introduced which reduce the complexity of proposed recognition framework. Initially, the frames from the input video is preprocessed. Next, the features exhibited by each emotions are extracted with geometric and local binary pattern-based feature extraction methods. Then, the features that reduce the performance of recognition technique is avoided using a feature selection algorithm. It selects the features that provides effective result on recognition process. Finally, the selected features are provided to deep belief network (DBN) for emotion recognition. The weight parameter selection of DBN is improved using an efficient Harris Hawk optimization algorithm. The performance of presented architecture is evaluated using a three different datasets they are FAMED, CK+, and MMI. The overall rate shown by proposed architecture is found better than existing methods. Furthermore, the precision, recall, and specificity are also evaluated for six different emotions (angry, disgust, fear, happy, sad, and surprise) in this proposed method. This entire emotion recognition process is implemented in Python platform.

show abstract

Bi-modality Fusion for Emotion Recognition in the Wild

Cited by 35 publications

References 15 publications

Pyramid With Super Resolution for In-the-Wild Facial Expression Recognition

Pyramid With Super Resolution for In-the-Wild Facial Expression Recognition

Context-Aware Emotion Recognition in the Wild Using Spatio-Temporal and Temporal-Pyramid Models

Modified deep belief network based human emotion recognition with multiscale features from video sequences

Contact Info

Product

Resources

About