Emotion Recognition from Facial Expression Using Hybrid CNN–LSTM Network

Abstract

Facial Expression Recognition (FER) is a prominent research area in Computer Vision and Artificial Intelligence that has been playing a crucial role in human–computer interaction. The existing FER system focuses on spatial features for identifying the emotion, which suffers when recognizing emotions from a dynamic sequence of facial expressions in real time. Deep learning techniques based on the fusion of convolutional neural networks (CNN) and long short-term memory (LSTM) are presented in this paper for recognizing emotion and identifying the relationship between the sequence of facial expressions. In this approach, a hyperparameter tweaked VGG-19 skeleton is employed to extract the spatial features automatically from a sequence of images, which avoids the shortcoming of the conventional feature extraction methods. Second, these features are given into bidirectional LSTM (Bi-LSTM) for extracting spatiotemporal features of time series in two directions, which recognize emotion from a sequence of expressions. The proposed method’s performance is evaluated using the CK+ benchmark as well as an in-house dataset captured from the designed IoT kit. Finally, this approach has been verified through hold-out cross-validation techniques. The proposed techniques show an accuracy of 0.92% on CK+, and 0.84% on the in-house dataset. The experimental results reveal that the proposed method outperforms compared to baseline methods and state-of-the-art approaches. Furthermore, precision, recall, F1-score, and ROC curve metrics have been used to evaluate the performance of the proposed system.

Context Transformer and Adaptive Method with Visual Transformer for Robust Facial Expression Recognition

“…SL+SSLpuzzling [51] 2021 98.23 FER_RN [49] 2022 96.97 CFNet [31] 2023 99.07 DBN [32] 2023 98.19 CNN_LSTM [52] 2023 92.00 ZFER [50] 2023 98.74 CoT_AdaptiveViT(Ours) 2024 99.20…”