Facial Emotion Recognition for Photo and Video Surveillance Based on Machine Learning and Visual Analytics

Abstract

Modern video surveillance systems mainly rely on human operators to monitor and interpret the behavior of individuals in real time, which may lead to severe delays in responding to an emergency. Therefore, there is a need for continued research into the designing of interpretable and more transparent emotion recognition models that can effectively detect emotions in safety video surveillance systems. This study proposes a novel technique incorporating a straightforward model for detecting sudden changes in a person’s emotional state using low-resolution photos and video frames from surveillance cameras. The proposed technique includes a method of the geometric interpretation of facial areas to extract features of facial expression, the method of hyperplane classification for identifying emotional states in the feature vector space, and the principles of visual analytics and “human in the loop” to obtain transparent and interpretable classifiers. The experimental testing using the developed software prototype validates the scientific claims of the proposed technique. Its implementation improves the reliability of abnormal behavior detection via facial expressions by 0.91–2.20%, depending on different emotions and environmental conditions. Moreover, it decreases the error probability in identifying sudden emotional shifts by 0.23–2.21% compared to existing counterparts. Future research will aim to improve the approach quantitatively and address the limitations discussed in this paper.

Explainable Deep Learning: A Visual Analytics Approach with Transition Matrices

“…In some VA tasks, particularly when applying the principle of HITL, there arises a need to determine the coefficients of the hyperplane from points that belong to or (more often) are near this hyperplane [29]. The determination of the hyperplane coefficients is proposed in a manner analogous to work [30].…”

Determinants of Emotion Recognition System Adoption: Empirical Evidence from Malaysia

“…Similarly, the use of facial expressions for ERS towards video surveillance was proposed in [38]. It has been highlighted that video surveillance systems nowadays are operated via human capabilities to interpret behavior through video surveillance, which leads to delays in responding to emergencies [38].…”

“…Similarly, the use of facial expressions for ERS towards video surveillance was proposed in [38]. It has been highlighted that video surveillance systems nowadays are operated via human capabilities to interpret behavior through video surveillance, which leads to delays in responding to emergencies [38]. The experiments concluded that the implementation of facial recognition as part of ERS towards video surveillance system can improve the reliability of abnormal behavior detection via facial expressions depending on different emotions and environmental conditions.…”

Three-Stage Deep Learning Framework for Video Surveillance

“…Research in this area, especially studies investigating surveillance video anomaly detection (SVAD) using deep learning [9], is progressing rapidly. This research domain has evolved to encompass a broader scope, extending beyond behavior-based detection to include advanced areas like human facial emotion recognition for anomaly detection [10], illustrating the expanding reach and depth of investigations in this field. These systems function by capturing real-time images across various indoor and outdoor settings, both public and private.…”