Social risk and depression: Evidence from manual and automatic facial expression analysis

Abstract

Investigated the relationship between change over time in severity of depression symptoms and facial expression. Depressed participants were followed over the course of treatment and video recorded during a series of clinical interviews. Facial expressions were analyzed from the video using both manual and automatic systems. Automatic and manual coding were highly consistent for FACS action units, and showed similar effects for change over time in depression severity. For both systems, when symptom severity was high, participants made more facial expressions associated with contempt, smiled less, and those smiles that occurred were more likely to be accompanied by facial actions associated with contempt. These results are consistent with the “social risk hypothesis” of depression. According to this hypothesis, when symptoms are severe, depressed participants withdraw from other people in order to protect themselves from anticipated rejection, scorn, and social exclusion. As their symptoms fade, participants send more signals indicating a willingness to affiliate. The finding that automatic facial expression analysis was both consistent with manual coding and produced the same pattern of depression effects suggests that automatic facial expression analysis may be ready for use in behavioral and clinical science.

Inferring mood in ubiquitous conversational video

“…Based on these techniques, the automatic analysis of spontaneous facial expressions in the wild is one key topic in affective computing. The target affective states include prototypical emotions [44], emotional dimensions such as valence and arousal [32], empathy [23], pain [29,27], and depression [19]. Some of them have focused on the observers' impressions about the target person [32,23], like the present study.…”

In the Mood for Vlog

“…The analysis of spontaneous facial expressions in the wild is currently a key topic in affective computing. The target affective states include prototypical emotions [Valstar et al 2011], emotional dimensions such as valence and arousal [McKeown et al 2010], empathy [Kumano et al 2012], pain [Lucey et al 2012;Littlewort et al 2007], and depression [Girard et al 2013]. Some works have focused on the observers' impressions about the target person [McKeown et al 2010;Kumano et al 2012], such as the present study.…”

Automatic Behaviour Understanding in Medicine

“…[35,8]. More recently, Girard et al [12] performed a longitudinal study of manual and automatic facial expressions during semistructured clinical interviews of 34 clinically depressed patients. They found that for both manual and automatic facial muscle activity analysis, participants with high symptom severity produced more expressions associated with contempt, smile less, and the smiles that were made were more likely to be related to contempt.…”