Utilizing computer vision for facial behavior analysis in schizophrenia studies: A systematic review

Abstract

Background Schizophrenia is a severe psychiatric disorder that causes significant social and functional impairment. Currently, the diagnosis of schizophrenia is based on information gleaned from the patient’s self-report, what the clinician observes directly, and what the clinician gathers from collateral informants, but these elements are prone to subjectivity. Utilizing computer vision to measure facial expressions is a promising approach to adding more objectivity in the evaluation and diagnosis of schizophrenia. Method We conducted a systematic review using PubMed and Google Scholar. Relevant publications published before (including) December 2021 were identified and evaluated for inclusion. The objective was to conduct a systematic review of computer vision for facial behavior analysis in schizophrenia studies, the clinical findings, and the corresponding data processing and machine learning methods. Results Seventeen studies published between 2007 to 2021 were included, with an increasing trend in the number of publications over time. Only 14 articles used interviews to collect data, of which different combinations of passive to evoked, unstructured to structured interviews were used. Various types of hardware were adopted and different types of visual data were collected. Commercial, open-access, and in-house developed models were used to recognize facial behaviors, where frame-level and subject-level features were extracted. Statistical tests and evaluation metrics varied across studies. The number of subjects ranged from 2-120, with an average of 38. Overall, facial behaviors appear to have a role in estimating diagnosis of schizophrenia and psychotic symptoms. When studies were evaluated with a quality assessment checklist, most had a low reporting quality. Conclusion Despite the rapid development of computer vision techniques, there are relatively few studies that have applied this technology to schizophrenia research. There was considerable variation in the clinical paradigm and analytic techniques used. Further research is needed to identify and develop standardized practices, which will help to promote further advances in the field.

Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns

“…To include facial behaviors less affected by cultural differences, we adopted JAA-Net [64] to recognize 49 facial landmarks and 12 facial action units [65] (AUs, or the individual components of facial muscle movement) expressed in the frame. JAA-Net is a deep learning model that combines CNN and adaptive attention module, and it achieved an average AU detection accuracy of 78.6% (including AU1, 2, 4, 6, 7, 10,12,14,15,17,23,24) and face alignment mean error of 3.8% inter-ocular distance on BP4D dataset [66] with threefold cross-validation.…”

“…The rapid development of objective automated digital assessment tools has the potential to aid clinicians in the diagnosis and evaluation of mental illness and to limit the impact of these illnesses on patients and on society [19]. Research groups have developed tools using various types of data modality, validated in numerous mental health populations, including depression [20], [21], anxiety disorder [22], schizophrenia [23], posttraumatic stress disorder (PTSD) [24], and almost all common psychiatric disorders. Diverse modalities of signals have been investigated, including behavioral signals, such as facial and body movements [20], [21], [25], speech acoustics [26]- [28], verbal or written content [29], sleep [30] and activity [24], [31] patterns, as well as physiological signals such as cardiovascular (heart rate [24], [32], electrocardiogram [28], [33], etc. )…”

Multimodal Assessment of Schizophrenia and Depression Utilizing Video, Acoustic, Locomotor, Electroencephalographic, and Heart Rate Technology: Protocol for an Observational Study

“…The development of easy-to-use, objective clinical tools to aid clinicians in the diagnosis and evaluation of mental illness has the potential to limit the impact of these illnesses on patients and on society. The cost of powerful computing hardware has fallen, and improvements in the field of computer science and health care suggest that various types of computer sensors and recording hardware could be used to aid in the assessment and diagnostic prediction of mental illness [25][26][27][28][29][30]. Research groups have demonstrated the efficacy in numerous mental health populations of these technologies, which include computer vision for distinguishing phases of depression [25,26], schizophrenia [27], and cognitive impairment [31], actigraphy for the differentiation of patients with schizophrenia from controls [28,29], and heart rate monitoring for distinguishing patients with schizophrenia or posttraumatic stress disorder from controls [28][29][30].…”

“…The cost of powerful computing hardware has fallen, and improvements in the field of computer science and health care suggest that various types of computer sensors and recording hardware could be used to aid in the assessment and diagnostic prediction of mental illness [25][26][27][28][29][30]. Research groups have demonstrated the efficacy in numerous mental health populations of these technologies, which include computer vision for distinguishing phases of depression [25,26], schizophrenia [27], and cognitive impairment [31], actigraphy for the differentiation of patients with schizophrenia from controls [28,29], and heart rate monitoring for distinguishing patients with schizophrenia or posttraumatic stress disorder from controls [28][29][30]. This research has demonstrated that with heart rate variability and actigraphic assessments alone, patients with schizophrenia may be differentiated from controls with up to 95.3% accuracy [28,29].…”

Evaluating and mitigating unfairness in multimodal remote mental health assessments

“…Objective automated digital assessment tools and corresponding digital biomarkers have been widely developed to aid clinicians in addressing the access, subjectivity, and bias challenges [15]. Those tools have been evaluated in various types of mental health disorders, using single data modality [16][17][18] or multiple modalities [19][20][21], in both lab-controlled [22] and remotely collected datasets [23,24]. Although promising results have been shown in those studies, the potential bias inherent in the proposed methodologies could impede the fair diagnosis and evaluation of the underprivileged or underrepresented groups, leading to detrimental health consequences.…”