Video-Based Stress Detection through Deep Learning

Abstract

Stress has become an increasingly serious problem in the current society, threatening mankind’s well-beings. With the ubiquitous deployment of video cameras in surroundings, detecting stress based on the contact-free camera sensors becomes a cost-effective and mass-reaching way without interference of artificial traits and factors. In this study, we leverage users’ facial expressions and action motions in the video and present a two-leveled stress detection network (TSDNet). TSDNet firstly learns face- and action-level representations separately, and then fuses the results through a stream weighted integrator with local and global attention for stress identification. To evaluate the performance of TSDNet, we constructed a video dataset containing 2092 labeled video clips, and the experimental results on the built dataset show that: (1) TSDNet outperformed the hand-crafted feature engineering approaches with detection accuracy 85.42% and F1-Score 85.28%, demonstrating the feasibility and effectiveness of using deep learning to analyze one’s face and action motions; and (2) considering both facial expressions and action motions could improve detection accuracy and F1-Score of that considering only face or action method by over 7%.

Deep-Learning-Based Stress Recognition with Spatial-Temporal Facial Information

“…The experimental results in Section 5. 4 show that the proposed method could detect overall spatial and temporal changes in the face related to stress and that it is superior to the method presented in the previous work [30].…”

“…All the methods introduced above used handcrafted features, but there was also a method using deep learning. This method recognizes stress by fusing facial images and motion information such as hand movements [30]. In this method, optical flow images were used to obtain motion information, and stress was recognized by applying attention to facial features and motion features.…”

“…If deep learning is used, the stress recognition performance can be further improved because stress-related high-dimensional features can be learned from images. Recently, a study [30] that recognized stress using deep learning came out, and we also tried to recognize stress using deep learning for better performance.…”

“…Some recent studies have recognized stress using only deep learning. Zhang et al [30] proposed a deep-learning-based method that detects the presence or absence of stress using the video footage of a person watching a video clip that induces or does not induce stress. In this method, when the face-level representation was first learned, an emotion recognition network was used to learn the emotion change between the two frames with the largest emotion difference.…”

Intelligent Stress Monitoring Assistant for First Responders

“…"Stress tracker" color bar on the Samsung Health app) are built on different principles; they rather approximately "estimate" the level of stress based on the heart rate measured by placing a finger on the sensor (if available on the phone) or via a Smartwatch. Limitation: Face is a useful source of data for stress detection [10], [11]; however, it may be occluded by the protective equipment. In contrast, in public-centric applications, e.g.…”