Identifying psychiatric manifestations in schizophrenia and depression from audio-visual behavioural indicators through a machine-learning approach

Abstract

Schizophrenia (SCZ) and depression (MDD) are two chronic mental disorders that seriously affect the quality of life of millions of people worldwide. We aim to develop machine-learning methods with objective linguistic, speech, facial, and motor behavioral cues to reliably predict the severity of psychopathology or cognitive function, and distinguish diagnosis groups. We collected and analyzed the speech, facial expressions, and body movement recordings of 228 participants (103 SCZ, 50 MDD, and 75 healthy controls) from two separate studies. We created an ensemble machine-learning pipeline and achieved a balanced accuracy of 75.3% for classifying the total score of negative symptoms, 75.6% for the composite score of cognitive deficits, and 73.6% for the total score of general psychiatric symptoms in the mixed sample containing all three diagnostic groups. The proposed system is also able to differentiate between MDD and SCZ with a balanced accuracy of 84.7% and differentiate patients with SCZ or MDD from healthy controls with a balanced accuracy of 82.3%. These results suggest that machine-learning models leveraging audio-visual characteristics can help diagnose, assess, and monitor patients with schizophrenia and depression.

Multimodal Assessment of Schizophrenia Symptom Severity From Linguistic, Acoustic and Visual Cues

“…Saga et al [8] directly adopted predefined features like frequency, bandwidth and volume of voice signals as acoustic features, which are then concatenated with FAU as input features for a random forest classifier to distinguish schizophrenia patients from healthy subjects. Xu et al [21] built an ensemble model based on linguistic, acoustic and visual low-level hand-crafted features, which is the first research incorporating tri-modality information into schizophrenia assessment. However, despite these automatic assessment systems have considered multimodal information, they are all based on relatively simple traditional machine learning techniques instead of deep learning which has led to the recent success in artificial intelligence.…”

“…Despite various methods have been proposed for automatic schizophrenia symptom assessment, most of them simply focus on one or two of the modalities, except only [21] which analyzed three kinds of modalities simultaneously. Furthermore, to the best of our knowledge, almost all the previous works are based on relatively simple traditional machine learning techniques instead of deep learning, which has led to the recent success in artificial intelligence.…”

Ethical considerations for integrating multimodal computer perception and neurotechnology

“…Investigations in this field range from animal studies to discover neuromarkers of animal behavior to studies with human subjects to explore neural markers of psychosocial and behavioral states, typically for the purposes of applied (rather than basic) clinical research (Torous et al, 2019). Examples include studies linking neural activity to vocal and sociobehavioral indicators of Parkinson's disease (Smith et al, 2017); facial and vocal markers of schizophrenia (Xu et al, 2022), and psychosocial markers of major depressive disorder (Mundt et al, 2012), with a goal of identifying personalized treatment (e.g., stimulation) approaches based on patient-specific symptom constellations and neural patterns. These efforts expand the repertoire of biometric and behavioral markers guided by a hope that neural markers may provide more direct representations of internal states such as emotions, cognition or intentions (Sheth et al, 2022).…”