Multimodal Measurement of Depression Using Deep Learning Models

Yang, Le; Jiang, Dongmei; Xia, Xiaohan; Pei, Ercheng; Oveneke, Meshia Cédric; Sahli, Hichem

doi:10.1145/3133944.3133948

Cited by 119 publications

(93 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, deep learning techniques that learn discriminant feature representations from training data are considered to be stateof-the-art in depression recognition. Deep learning models for video-based depression detection often cascade a 2D CNN with an RNN [17]. Most notably, Jan et al [38] propose deep learning techniques to extract features from facial frames and employ feature dynamic history histogram (FDHH) to capture variations in the features.…”

Section: Related Workmentioning

confidence: 99%

“…Deep learning architectures, and in particular CNNs, provide state-of-the-art performance in many visual recognition applications, such as image classification [14] and object detection [15], as well as assisted medical diagnosis [16]. In depression detection, deep learning architectures that process on videos typically exploit spatial and temporal information separately (e.g., by cascading a 2D CNN and then a recurrent NN), which deteriorate the modeling of spatio-temporal relationships [11], [17]. A deep two-stream architecture has also been proposed to exploit facial appearance and facial optical flow [10].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Combining Global and Local Convolutional 3D Networks for Detecting Depression from Facial Expressions

Melo

Granger

Hadid

2019

2019 14th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2019)

View full text Add to dashboard Cite

Deep learning architectures have been successfully applied in video-based health monitoring, to recognize distinctive variations in the facial appearance of subjects. To detect patterns of variation linked to depressive behavior, deep neural networks (NNs) typically exploit spatial and temporal information separately by, e.g., cascading a 2D convolutional NN (CNN) with a recurrent NN (RNN), although the intrinsic spatio-temporal relationships can deteriorate. With the recent advent of 3D CNNs like the convolutional 3D (C3D) network, these spatio-temporal relationships can be modeled to improve performance. However, the accuracy of C3D networks remain an issue when applied to depression detection. In this paper, the fusion of diverse C3D predictions are proposed to improve accuracy, where spatio-temporal features are extracted from global (full-face) and local (eyes) regions of subject. This allows to increasingly focus on a local facial region that is highly relevant for analyzing depression. Additionally, the proposed network integrates 3D Global Average Pooling in order to efficiently summarize spatio-temporal features without using fully-connected layers, and thereby reduce the number of model parameters and potential over-fitting. Experimental results on the Audio Visual Emotion Challenge (AVEC 2013 and AVEC 2014) depression datasets indicates that combining the responses of global and local C3D networks achieves a higher level of accuracy than state-of-the-art systems.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Combining Global and Local Convolutional 3D Networks for Detecting Depression from Facial Expressions

Melo

Granger

Hadid

2019

2019 14th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2019)

View full text Add to dashboard Cite

show abstract

“…In a similar spirit, and Sun et al modeled depression by structuring questions and responses in the form a decision tree [7,8], while Gong et al developed an ensemble of audio, text, and video features as a function of the question type asked [9]. Utilizing a deep learning framework, Yang et al (2017) combined multiple modalities conditioned on manually selected questions [10].…”

Section: Introductionmentioning

confidence: 99%

Detecting Depression with Audio/Text Sequence Modeling of Interviews

Alhanai¹,

2018

View full text Add to dashboard Cite

Medical professionals diagnose depression by interpreting the responses of individuals to a variety of questions, probing lifestyle changes and ongoing thoughts. Like professionals, an effective automated agent must understand that responses to queries have varying prognostic value. In this study we demonstrate an automated depression-detection algorithm that models interviews between an individual and agent and learns from sequences of questions and answers without the need to perform explicit topic modeling of the content. We utilized data of 142 individuals undergoing depression screening, and modeled the interactions with audio and text features in a Long-Short Term Memory (LSTM) neural network model to detect depression. Our results were comparable to methods that explicitly modeled the topics of the questions and answers which suggests that depression can be detected through sequential modeling of an interaction, with minimal information on the structure of the interview.

show abstract

“…Although the use of different scales for training (e.g., Yang et al 2017 used PHQ-8 scores) can introduce potential confounds for performance comparisons, we scaled our model's MAE and RMSE values to obtain an error percentage. When we compared error percentages, we found that our model fared better in terms of MAE compared to previous models and performed comparably with respect to RMSE percentage (L. Yang et al 2017). It is noteworthy that this difference in performance might arise because of a difference in the distribution of the original scores (whether the datasets contain more depressed participants).…”

Section: Resultsmentioning

confidence: 68%

“…In parallel, ML techniques have been applied to examine affective display differences exhibited during emotion states, such as facial expression and vocal prosody, through audio and video-based analysis. These advances have generated a new field of research which has successfully used ML techniques, such as support vector machines (Cohn et al 2009), regression (Valstar et al 2013), and neural networks (shallow and deep; (L. Yang et al 2017)), for automatic recognition of emotion using audiovisual data from conventional databases (Schuller, Steidl, and Batliner 2009;Burkhardt et al 2005) and recently more naturalistic environments (Dhall et al 2013;McKeown et al 2012;Ringeval et al 2013). Moreover, ML has also been extended to investigate verbal and nonverbal affective abnormalities associated with psychiatric disorders and has gone on to successfully classify those presenting with and without a given diagnosis (Hamm et al 2011;P.…”

Section: Introductionmentioning

confidence: 99%

Detecting depression using a framework combining deep multimodal neural networks with a purpose-built automated evaluation.

Victor¹,

Aghajan²,

Sewart³

et al. 2019

Psychological Assessment

View full text Add to dashboard Cite

Machine learning (ML) has been introduced into the medical field as a means to provide diagnostic tools capable of enhancing accuracy and precision while minimizing laborious tasks that require human intervention. There is mounting evidence that the technology fueled by ML has the potential to detect, and substantially improve treatment of complex mental disorders such as depression. We developed a framework capable of detecting depression with minimal human intervention: AiME (Artificial Intelligence Mental Evaluation). AiME consists of a short human-computer interactive evaluation and artificial intelligence, namely deep learning, and can predict whether the participant is depressed or not with satisfactory performance. Due to its ease of use, this technology can offer a viable tool for mental health professionals to identify symptoms of depression, thus enabling a faster preventative intervention. Furthermore, it may alleviate the challenge of interpreting highly nuanced physiological and behavioral biomarkers of depression by providing a more objective evaluation.

show abstract

Multimodal Measurement of Depression Using Deep Learning Models

Cited by 119 publications

References 20 publications

Combining Global and Local Convolutional 3D Networks for Detecting Depression from Facial Expressions

Combining Global and Local Convolutional 3D Networks for Detecting Depression from Facial Expressions

Detecting Depression with Audio/Text Sequence Modeling of Interviews

Detecting depression using a framework combining deep multimodal neural networks with a purpose-built automated evaluation.

Contact Info

Product

Resources

About