Video-Based Facial Expression Recognition using Deep Temporal–Spatial Networks

Pan, Xianzhang; Zhang, Shiqing; Guo, Wenping; Zhao, Xiaoming; Chuang, Yuelong; Chen, Ying; Zhang, Haibo

doi:10.1080/02564602.2019.1645620

Cited by 12 publications

(3 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fan et al [23] fused discriminative features extracted by CNN model with features containing shape and appearance extracted by hand. Pan et al [24] designed a Deep Temporal-Spatial Network based on facial expressions to extract the spatiotemporal features. Chen et al [25] proposed a facial feature called deep peak-calmness difference (DPND), which can characterize facial regions that change from calmness to expressive face, and achieved high-quality results in both unsupervised clustering and semi-supervised classification methods.…”

Section: A Emotion Recognition With Facial Expressionsmentioning

confidence: 99%

Emotion Recognition of Subjects With Hearing Impairment Based on Fusion of Facial Expression and EEG Topographic Map

Liu

Yang

et al. 2023

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

Emotion analysis has been employed in many fields such as human-computer interaction, rehabilitation, and neuroscience. But most emotion analysis methods mainly focus on healthy controls or depression patients. This paper aims to classify the emotional expressions in individuals with hearing impairment based on EEG signals and facial expressions. Two kinds of signals were collected simultaneously when the subjects watched affective video clips, and we labeled the video clips with discrete emotional states (fear, happiness, calmness, and sadness). We extracted the differential entropy (DE) features based on EEG signals and converted DE features into EEG topographic maps (ETM). Next, the ETM and facial expressions were fused by the multichannel fusion method. Finally, a deep learning classifier CBAM_ResNet34 combined Residual Network (ResNet) and Convolutional Block Attention Module (CBAM) was used for subject-dependent emotion classification. The results show that the average classification accuracy of four emotions recognition after multimodal fusion achieves 78.32%, which is higher than 67.90% for facial expressions and 69.43% for EEG signals. Moreover, visualization by the Gradient-weighted Class Activation Mapping (Grad-CAM) of ETM showed that the prefrontal, temporal and occipital lobes were the brain regions closely related to emotional changes in individuals with hearing impairment.

show abstract

Section: A Emotion Recognition With Facial Expressionsmentioning

confidence: 99%

Emotion Recognition of Subjects With Hearing Impairment Based on Fusion of Facial Expression and EEG Topographic Map

Liu

Yang

et al. 2023

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

show abstract

“…English education resource libraries of Arduino device images collected on-site are selected as experimental data, including Arduino UNO board, L298 N drive board, Hall sensor, active buzzer, rocker, serial wireless transparent transmission module, PIR human body sensor, potential device, and ultrasonic module and LCD [18,19]. Each device collects 500 images, and a total of 5000 images are collected.…”

Section: Data Sources and Preprocessing In This Experiment 10mentioning

confidence: 99%

Research on Optimization and Allocation of English Teaching Resources

Liu

Xia

2022

Mathematical Problems in Engineering

View full text Add to dashboard Cite

To rationally allocate teaching resources in English teaching, a teaching resource optimization and allocation management method is proposed based on a convolutional neural network (CNN) and Arduino device. By constructing a 9-layer CNN classification and recognition model and the English education resource library of the Arduino device and applying them to the recognition program design of the Arduino device, the rational optimization and allocation of teaching resources are realized. Simulation results show that the recognition accuracy of the proposed method is over 90% for Arduino devices, and the recognition accuracy is over 80% for real English teaching scenarios, which means that the proposed method has a certain practical application value. Moreover, the interaction mode between English learners and English teaching resources is innovated, which contributes to the optimization and allocation of English teaching resources. Thus a new idea is generated to integrate the English teaching resources.

show abstract

“…They also proposed the use of average human face as a substitute to the neutral face for the cases where neutral face was not available for reference. Pan et al [41] utilized the magnitude of the optical flow between successive frames in a video to characterize their relative motion as a form of a temporal channel in their spatiotemporal video-based FER model.…”

Section: Optical Flow and Fermentioning

confidence: 99%

Modelling and Analysis of Facial Expressions Using Optical Flow Derived Divergence and Curl Templates

Anthwal

2021

ELCVIA

View full text Add to dashboard Cite

Facial expressions are integral part of non-verbal paralinguistic communication as they provide cues significant in perceiving one’s emotional state. Assessment of emotions through expressions is an active research domain in computer vision due to its potential applications in multi-faceted domains. In this work, an approach is presented where facial expressions are modelled and analyzed with dense optical flow derived divergence and curl templates that embody the ideal motion pattern of facial features pertaining to unfolding of an expression on the face. Two types of classification schemes based on multi-class support vector machine and k-nearest neighbour are employed for evaluation. Promising results obtained from comparative analysis of the proposed approach with state-of-the-art techniques on the Extended Cohn Kanade database and with human cognition and pre-trained Microsoft face application programming interface on the Karolinska Directed Emotional Faces database validate the efficiency of the approach.

show abstract

Video-Based Facial Expression Recognition using Deep Temporal–Spatial Networks

Cited by 12 publications

References 20 publications

Emotion Recognition of Subjects With Hearing Impairment Based on Fusion of Facial Expression and EEG Topographic Map

Emotion Recognition of Subjects With Hearing Impairment Based on Fusion of Facial Expression and EEG Topographic Map

Research on Optimization and Allocation of English Teaching Resources

Modelling and Analysis of Facial Expressions Using Optical Flow Derived Divergence and Curl Templates

Contact Info

Product

Resources

About