Measuring depression severity based on facial expression and body movement using deep convolutional neural network

Abstract

IntroductionReal-time evaluations of the severity of depressive symptoms are of great significance for the diagnosis and treatment of patients with major depressive disorder (MDD). In clinical practice, the evaluation approaches are mainly based on psychological scales and doctor-patient interviews, which are time-consuming and labor-intensive. Also, the accuracy of results mainly depends on the subjective judgment of the clinician. With the development of artificial intelligence (AI) technology, more and more machine learning methods are used to diagnose depression by appearance characteristics. Most of the previous research focused on the study of single-modal data; however, in recent years, many studies have shown that multi-modal data has better prediction performance than single-modal data. This study aimed to develop a measurement of depression severity from expression and action features and to assess its validity among the patients with MDD.MethodsWe proposed a multi-modal deep convolutional neural network (CNN) to evaluate the severity of depressive symptoms in real-time, which was based on the detection of patients’ facial expression and body movement from videos captured by ordinary cameras. We established behavioral depression degree (BDD) metrics, which combines expression entropy and action entropy to measure the depression severity of MDD patients.ResultsWe found that the information extracted from different modes, when integrated in appropriate proportions, can significantly improve the accuracy of the evaluation, which has not been reported in previous studies. This method presented an over 74% Pearson similarity between BDD and self-rating depression scale (SDS), self-rating anxiety scale (SAS), and Hamilton depression scale (HAMD). In addition, we tracked and evaluated the changes of BDD in patients at different stages of a course of treatment and the results obtained were in agreement with the evaluation from the scales.DiscussionThe BDD can effectively measure the current state of patients’ depression and its changing trend according to the patient’s expression and action features. Our model may provide an automatic auxiliary tool for the diagnosis and treatment of MDD.

Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

“…There were also novel approaches adapting various concepts, including recommender system (RS) [ 173 , 307 ], node classification [ 173 ], and federated learning [ 168 ]. Additionally, some studies employed computations to learn association parameters [ 83 , 189 ] or deduce prediction outcomes from distance-based homogeneity [ 85 ].…”

MoodCapture: Depression Detection using In-the-Wild Smartphone Images

“…Extracting facial features to assess mental health and emotions has received significant attention in computer vision, with applications spanning from education to healthcare [51]. Here, many studies have explored facial expressions, gaze patterns, and the overall composition of images to extract visual markers symptomatic of depression [38,43,46]. However, most of these studies are conducted in controlled environments or rely on participants deliberately capturing their images, which could inadvertently influence their emotional portrayal.…”

“…Participants were asked to sit before a white background, remove hats or glasses, and tie up long hair to expose their ears; the users looked straight ahead with relaxed expressions as instructed. Similarly, Liu et al [46] employed a multi-modal deep Convolutional Neural Network (CNN), considering both facial expressions and body movements.…”

“…Most of the prior research utilizing facial images to detect depression focus on capturing these images in controlled settings, where individuals may be instructed to perform specific actions [22,38,43,46]. These face features are not authentic as they are performative and are influenced by biases such as social desirability and self-presentation.…”

A Dynamic Probabilistic Model for Heterogeneous Data Fusion: A Pilot Case Study from Computer-Aided Detection of Depression

“…More precisely, in this case, for any subject, the experimental procedure involves the collection of speech data obtained from the reading of a narrative text and the free description of daily activities performed during a week, and the answers provided by the participants to data collection to the questions of BDI-II [ 4 ]. Further data can be collected on this line of conduct, other than speech, including facial expressions, handwriting and drawing, body movements, hands’ gestures, and any other activity useful for behavior analysis ([ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ], among others). Though all the outcomes of these experiments can be (and actually are) given a quantitative expression, by means of suitably defined measures or scores, nevertheless, this raises a new problem, regarding the comparison of heterogeneous measures.…”