Facial Expression Recognition in Video with Multiple Feature Fusion

Chen, Junkai; Chen, Zenghai; Chi, Zheru; Hong, Fei

doi:10.1109/taffc.2016.2593719

Cited by 184 publications

(85 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The overall system accuracy of 47.44% is comparable with similar state-of-the-art work of J. Chen, conducted on the extended Cohn-Kanade (CK+) database (clean environment) and the Acted Facial Expression in Wild (AFEW) 4.0 database [42], with an overall average accuracy of 45.2% and between 0.6% for clean and 5% for "wild" videos for the accuracy achievement, and using a new proposed histogram of oriented gradients from three orthogonal planes (HOG-TOP) compared to LBP features used in this study for real-world videos (close to wild). The objective of this study was to appraise the efficiency of regression methods for automatic emotional state detection and analysis on embedded devices to be made on human emotion recognition from live cameras.…”

Section: Conclusion and Discussionsupporting

confidence: 71%

Facial Expression Emotion Detection for Real-Time Embedded Systems

et al. 2018

View full text Add to dashboard Cite

Abstract:Recently, real-time facial expression recognition has attracted more and more research. In this study, an automatic facial expression real-time system was built and tested. Firstly, the system and model were designed and tested on a MATLAB environment followed by a MATLAB Simulink environment that is capable of recognizing continuous facial expressions in real-time with a rate of 1 frame per second and that is implemented on a desktop PC. They have been evaluated in a public dataset, and the experimental results were promising. The dataset and labels used in this study were made from videos, which were recorded twice from five participants while watching a video. Secondly, in order to implement in real-time at a faster frame rate, the facial expression recognition system was built on the field-programmable gate array (FPGA). The camera sensor used in this work was a Digilent VmodCAM -stereo camera module. The model was built on the Atlys TM Spartan-6 FPGA development board. It can continuously perform emotional state recognition in real-time at a frame rate of 30. A graphical user interface was designed to display the participant's video in real-time and two-dimensional predict labels of the emotion at the same time.

show abstract

Section: Conclusion and Discussionsupporting

confidence: 71%

Facial Expression Emotion Detection for Real-Time Embedded Systems

et al. 2018

View full text Add to dashboard Cite

show abstract

“…It is important to extract the features from the faces for the global alignment kernel, as the appropriate feature can better measure the distances of faces. The existing approaches on multiple feature fusion [43], [44] show using multiple features can use the benefit of different features and obtain the better performance than using sole feature. Additionally, the existing group-level emotion recognition databases were collected from the Internet and suffer from the noise caused by poor illumination, head pose and bad image quality.…”

Section: Svm Based On Combined Global Alignment Kernelmentioning

confidence: 99%

Analyzing Group-Level Emotion with Global Alignment Kernel based Approach

Huang

Dhall

Goecke

et al. 2022

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

From the perspective of social science, understanding group emotion has become increasingly important for teams to considerably accomplish organizational work. Currently, automatically analyzing the perceived affect of a group of people has been received increasingly interest in affective computing community. The variability in group size makes difficulty for group-level emotion recognition to straightforwardly measure the feature distance of two group-level images. Recent works attempted to resolve the preceding problem by using feature encoding. However, the early works lack of efficiency. To alleviate this problem, this paper aims to design a new method to effectively analyze the group behavior from a group-level image. Motivated by time-series kernel approaches explored in dynamic facial expression classification, this paper mainly concentrates on global alignment kernel and design support vector machine with the combined global alignment kernels (SVM-CGAK) to better recognize group-level emotion. Specifically, we first propose to use global alignment kernel to explicitly measure the distance of two group-level images. For improving the performance of global alignment kernel, we use the global weight sort scheme based on their spatial relation information to sort the faces from group-level image, making an efficient data structure to the global alignment kernel. With this new global alignment kernel, we construct the backbone of SVM-CGAK, namely, support vector machine with global alignment kernel. Furthermore, considering the challenging environment, we construct two global alignment kernels based on Reisz-based Volume Local Binary Pattern and deep convolutional neural network features, respectively. Lastly, to make the robustness of group-level emotion recognition, we propose SVM-CGAK combining both global alignment kernels with multiple kernel learning approach. It can enhance the discriminative ability of each global alignment kernel. Intensive experiments are conducted on three challenging group-level emotion databases. The experimental results demonstrate that the proposed approach achieves promising performance for group-level emotion recognition compared with the recent state-of-the-art methods.

show abstract

“…Step 1: The Gini index is used to measure the segmentation effect of a feature in a decision tree In this paper, to train the RF algorithm, we extract 106 active patches form each micro-expression and calculate the joint histogram integrated by the optical flow feature and LBP-TOP operator of each patch. Ultimately, they form the 106-feature histogram vectors as shown in Equation (19).…”

Section: Nmp Extraction From the Active Patchesmentioning

confidence: 99%

An Improved Micro-Expression Recognition Method Based on Necessary Morphological Patches

Zhao

Jia-dong

2019

Symmetry

View full text Add to dashboard Cite

Micro-expression is a spontaneous emotional representation that is not controlled by logic. A micro-expression is both transitory (short duration) and subtle (small intensity), so it is difficult to detect in people. Micro-expression detection is widely used in the fields of psychological analysis, criminal justice and human-computer interaction. Additionally, like traditional facial expressions, micro-expressions also have local muscle movement. Psychologists have shown micro-expressions have necessary morphological patches (NMPs), which are triggered by emotion. Furthermore, the objective of this paper is to sort and filter these NMPs and extract features from NMPs to train classifiers to recognize micro-expressions. Firstly, we use the optical flow method to compare the on-set frame and the apex frame of the micro-expression sequences. By doing this, we could find facial active patches. Secondly, to find the NMPs of micro-expressions, this study calculates the local binary pattern from three orthogonal planes (LBP-TOP) operators and cascades them with optical flow histograms to form the fusion features of the active patches. Finally, a random forest feature selection (RFFS) algorithm is used to identify the NMPs and to characterize them via support vector machine (SVM) classifier. We evaluated the proposed method on two popular publicly available databases: CASME II and SMIC. Results show that NMPs are statistically determined and contribute to significant discriminant ability instead of holistic utilization of all facial regions.

show abstract

Facial Expression Recognition in Video with Multiple Feature Fusion

Cited by 184 publications

References 42 publications

Facial Expression Emotion Detection for Real-Time Embedded Systems

Facial Expression Emotion Detection for Real-Time Embedded Systems

Analyzing Group-Level Emotion with Global Alignment Kernel based Approach

An Improved Micro-Expression Recognition Method Based on Necessary Morphological Patches

Contact Info

Product

Resources

About