SchiNet: Automatic Estimation of Symptoms of Schizophrenia from Facial Behaviour Analysis

Abstract

Patients with schizophrenia often display impairments in the expression of emotion and speech and those are observed in their facial behaviour. Automatic analysis of patients' facial expressions that is aimed at estimating symptoms of schizophrenia has received attention recently. However, the datasets that are typically used for training and evaluating the developed methods, contain only a small number of patients (4-34) and are recorded while the subjects were performing controlled tasks such as listening to life vignettes, or answering emotional questions. In this paper, we use videos of professional-patient interviews, in which symptoms were assessed in a standardised way as they should/may be assessed in practice, and which were recorded in realistic conditions (i.e. varying illumination levels and camera viewpoints) at the patients' homes or at mental health services. We automatically analyse the facial behaviour of 91 out-patients -this is almost 3 times the number of patients in other studies -and propose SchiNet, a novel neural network architecture that estimates expression-related symptoms in two different assessment interviews. We evaluate the proposed SchiNet for patient-independent prediction of symptoms of schizophrenia. Experimental results show that some automatically detected facial expressions are significantly correlated to symptoms of schizophrenia, and that the proposed network for estimating symptom severity delivers promising results.

Can Automatic Facial Expression Analysis Be Used for Treatment Outcome Estimation in Schizophrenia?

“…In order to test how SSE methods perform on Treatment Outcome Estimation (TOE), these methods are applied for estimating the symptom severity before and after treatment independently, and then the results are compared so as to reach a conclusion on the treatment outcome. We compare our architecture with three methods that have been proposed for SSE in schizophrenia [7,8,16] -for a fair comparison, we have re-implemented and re-trained those methods using the 74 patients. We used the probabilities of the 11 detected expressions in the training and the LOSO protocol for training/testing.…”

“…Diagnosis of schizophrenia. In [7,8,16], AFEA was used for the diagnosis or/and symptom severity estimation in schizophrenia. In [7,8], a commercial 3D facial analysis tool was applied for extracting 23 AUs.…”

“…In [7,8], the subjects were recorded while answering emotionally evocative questions. In [16], Bishay et al proposed to turn the analysis from controlled contexts and environments to settings that are similar to the ones found in clinics. Furthermore, a deep architecture (called SchiNet) was proposed for analysing patients' facial expressions in the wild, as well as estimating symptom severity by extracting deep statistical features using Gaussian mixture model and Fisher Vector layers.…”

“…in real world conditions) [20]. In this work we use the AFEA methods proposed in [16] and in [19] -while the specific choice of the AFEA method is not crucial, it is important that it can deal with illumination, pose and scale variations as those are common in the data that we have. For each frame in the clips, we get an 11dimensional feature vector, 10 dimensions corresponding to the probabilities of the presence of 10 facial action units (i.e.…”

“…For each frame in the clips, we get an 11dimensional feature vector, 10 dimensions corresponding to the probabilities of the presence of 10 facial action units (i.e. facial expressions) [16], and one corresponding to the probability of the presence of a smile [19].…”