A multimodal computer-aided diagnostic system for depression relapse prediction using audiovisual cues: A proof of concept

Othmani, Alice; Zeghina, Assaad Oussama

doi:10.1016/j.health.2022.100090

Cited by 16 publications

(4 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this manuscript, we proposed a multimodal deep convolution neural network to detect the severity of depression. Compared to an existing multimodal model ( 20 , 31 ), our model has the following improvements: (i) we integrated facial expression information that is an important feature in evaluating the severity of depression; (ii) we constructed the BDD metrics to quantify the severity of depression and achieved a good performance; (iii) we found that the information extracted from different modes, when integrated in appropriate proportions, can significantly improve the accuracy of the evaluation, which has not been reported in previous studies.…”

Section: Discussionmentioning

confidence: 99%

“…Most of the previous AI models focus on single-modal information of a certain feature of the patient. In recent years, some studies have shown that multi-modal information has a better prediction performance than single-modal data ( 20 , 31 ). In this paper, we proposed a multi-modal deep convolutional neural network (CNN) model based on facial expressions and body movements to evaluate the severity of depression.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Measuring depression severity based on facial expression and body movement using deep convolutional neural network

Liu

Lin

et al. 2022

Front. Psychiatry

View full text Add to dashboard Cite

IntroductionReal-time evaluations of the severity of depressive symptoms are of great significance for the diagnosis and treatment of patients with major depressive disorder (MDD). In clinical practice, the evaluation approaches are mainly based on psychological scales and doctor-patient interviews, which are time-consuming and labor-intensive. Also, the accuracy of results mainly depends on the subjective judgment of the clinician. With the development of artificial intelligence (AI) technology, more and more machine learning methods are used to diagnose depression by appearance characteristics. Most of the previous research focused on the study of single-modal data; however, in recent years, many studies have shown that multi-modal data has better prediction performance than single-modal data. This study aimed to develop a measurement of depression severity from expression and action features and to assess its validity among the patients with MDD.MethodsWe proposed a multi-modal deep convolutional neural network (CNN) to evaluate the severity of depressive symptoms in real-time, which was based on the detection of patients’ facial expression and body movement from videos captured by ordinary cameras. We established behavioral depression degree (BDD) metrics, which combines expression entropy and action entropy to measure the depression severity of MDD patients.ResultsWe found that the information extracted from different modes, when integrated in appropriate proportions, can significantly improve the accuracy of the evaluation, which has not been reported in previous studies. This method presented an over 74% Pearson similarity between BDD and self-rating depression scale (SDS), self-rating anxiety scale (SAS), and Hamilton depression scale (HAMD). In addition, we tracked and evaluated the changes of BDD in patients at different stages of a course of treatment and the results obtained were in agreement with the evaluation from the scales.DiscussionThe BDD can effectively measure the current state of patients’ depression and its changing trend according to the patient’s expression and action features. Our model may provide an automatic auxiliary tool for the diagnosis and treatment of MDD.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Measuring depression severity based on facial expression and body movement using deep convolutional neural network

Liu

Lin

et al. 2022

Front. Psychiatry

View full text Add to dashboard Cite

show abstract

“…Fully connected (FC) layers, multilayer perceptron (MLP), CNN, LSTM, BiLSTM, GRU, temporal convolutional network (TCN) [441] (with dilation for long sequences)-with activation function like Sigmoid, Softmax, ReLU, LeakyReLU, and GeLU Predict scores of assessment scales (regression) or probability distribution over classes (classification) [60,78,80,[84][85][86][87][88][90][91][92][93][94]96,98,105,111,113,117,131,133,135,136,[142][143][144]146,162,163,167,168,170,172,174,178,179,190,197,199,201,218,219,221,223,308] DCNN-DNN (combination of deep CNN and DNN), GCNN-LSTM (combination of gated convolutional neural network, which replaces a convolution block in CNN wi...…”

Section: Neural Networkmentioning

confidence: 99%

Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

Khoo,

Lim,

Chong

et al. 2024

Sensors

View full text Add to dashboard Cite

As mental health (MH) disorders become increasingly prevalent, their multifaceted symptoms and comorbidities with other conditions introduce complexity to diagnosis, posing a risk of underdiagnosis. While machine learning (ML) has been explored to mitigate these challenges, we hypothesized that multiple data modalities support more comprehensive detection and that non-intrusive collection approaches better capture natural behaviors. To understand the current trends, we systematically reviewed 184 studies to assess feature extraction, feature fusion, and ML methodologies applied to detect MH disorders from passively sensed multimodal data, including audio and video recordings, social media, smartphones, and wearable devices. Our findings revealed varying correlations of modality-specific features in individualized contexts, potentially influenced by demographics and personalities. We also observed the growing adoption of neural network architectures for model-level fusion and as ML algorithms, which have demonstrated promising efficacy in handling high-dimensional features while modeling within and cross-modality relationships. This work provides future researchers with a clear taxonomy of methodological approaches to multimodal detection of MH disorders to inspire future methodological advancements. The comprehensive analysis also guides and supports future researchers in making informed decisions to select an optimal data source that aligns with specific use cases based on the MH disorder of interest.

show abstract

“…First, a disproportionately large number of models are derived from small datasets, such as the Distress Analysis Interview Corpus Wizard-of-Oz (DAIC-WOZ), which contains only 189 examples (Gratch et al, 2014). Because it is one of the only publicly available datasets that contains both audiovisual recordings and depression-scale scores, researchers continue to use it for training and validating extremely complex ML models (Othmani & Zeghina, 2022) that likely require exponentially larger sample sizes to achieve out-of-sample generalizability (McNamara et al, 2022). Moreover, models based on audiovisual features have typically not accounted for the confound that depression is more prevalent among women, and have inadvertently used "vocal biomarkers" linked to sex assigned at birth to artificially boost their accuracy at predicting depression (Bailey & Plumbley, 2021).…”

mentioning

confidence: 99%

Conversational Assessment Using Artificial Intelligence is as Clinically Useful as Depression Scales and Preferred by Users

Weisenburger,

Mullarkey,

Labrada

et al. 2023

Preprint

View full text Add to dashboard Cite

Background: Depression is prevalent, chronic, and burdensome. Due to limited screening access, depression often remains undiagnosed. Artificial intelligence (AI) models based on spoken responses to interview questions may offer an effective, efficient alternative to other screening methods.Objective: The primary aim was to use a demographically diverse sample to validate an AI model, previously trained on human-administered interviews, on novel bot-administered interviews, and to check for algorithmic biases related to age, sex, race, and ethnicity.Methods: Using the Aiberry app, adults recruited via social media (N = 393) completed a brief bot-administered interview and a depression self-report form. An AI model was used to predict form scores based on interview responses alone. For all meaningful discrepancies between model inference and form score, clinicians performed a masked review to determine which one they preferred.Results: There was strong concurrent validity between the model predictions and raw self-report scores (r = 0.73, MAE = 3.3). 90% of AI predictions either agreed with self-report or with clinical expert opinion when AI contradicted self-report. There was no differential model performance across age, sex, race, or ethnicity.Limitations: Limitations include access restrictions (English-speaking ability and access to smartphone or computer with broadband internet) and potential self-selection of participants more favorably predisposed toward AI technology.Conclusion: The Aiberry model made accurate predictions of depression severity based on remotely collected spoken responses to a bot-administered interview. This study shows promising results for the use of AI as a mental health screening tool.

show abstract

A multimodal computer-aided diagnostic system for depression relapse prediction using audiovisual cues: A proof of concept

Cited by 16 publications

References 32 publications

Measuring depression severity based on facial expression and body movement using deep convolutional neural network

Measuring depression severity based on facial expression and body movement using deep convolutional neural network

Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

Conversational Assessment Using Artificial Intelligence is as Clinically Useful as Depression Scales and Preferred by Users

Contact Info

Product

Resources

About