Multimodal Assessment of Schizophrenia and Depression Utilizing Video, Acoustic, Locomotor, Electroencephalographic, and Heart Rate Technology: Protocol for an Observational Study

Cotes, Robert O.; Boazak, Mina; Griner, Emily; Jiang, Zifan; Kim, Bona; Bremer, Whitney; Seyedi, Salman; Rad, Ali Bahrami; Clifford, Gari D.

doi:10.2196/36417

Cited by 15 publications

(10 citation statements)

References 75 publications

(86 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, the number of subjects (n=73) and their heterogeneity might limit our findings’ generalizability. While the number of subjects will grow as we keep collecting data following our previous protocol [19], the heterogeneity issue might not be easily addressed. Although we recruited subjects with clear inclusion/exclusion criteria and further excluded subjects after the interview if they did not fall into our criteria, the intrinsic nature of high comorbidity levels in different mental disorders makes it difficult to recruit a “clean” cohort of subjects with clear diagnoses of a single type of disorder.…”

Section: Discussionmentioning

confidence: 99%

“…The study team created the interview guide and protocol and have components that simulate a psychiatric intake interview [19]. All interviews were conducted remotely via Zoom’s secure, encrypted, HIPAA-compliant telehealth platform.…”

Section: Datasetmentioning

confidence: 99%

“…The increasing use of telemedicine in psychiatry in recent years, even further accelerated by the COVID-19 pandemic [44], provided a promising approach to improve the access and effect of psychiatric care [45]- [47], while at the same presented an unprecedented opportunity of data collection for objective psychiatric assessments development without the limitation of geographical location and specialized hardware [48]. This begs the question of whether data collected remotely, such as in [49], [50], and in our previous research protocol [19], can provide a similar level of information as the data collected in a lab-controlled environment.…”

Section: Introductionmentioning

confidence: 99%

“…The rapid development of objective automated digital assessment tools has the potential to aid clinicians in the diagnosis and evaluation of mental illness and to limit the impact of these illnesses on patients and on society [19]. Research groups have developed tools using various types of data modality, validated in numerous mental health populations, including depression [20], [21], anxiety disorder [22], schizophrenia [23], posttraumatic stress disorder (PTSD) [24], and almost all common psychiatric disorders.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns

Jiang,

Seyedi,

Griner

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

ObjectiveThe current clinical practice of psychiatric evaluation suffers from subjectivity and bias, and requires highly skilled professionals that are often unavailable or unaffordable. Objective digital biomarkers have shown the potential to address these issues. In this work, we investigated whether behavioral and physiological signals, extracted from remote interviews, provided complimentary information for assessing psychiatric disorders.MethodsTime series of multimodal features were derived from four conceptual modes: facial expression, vocal expression, linguistic expression, and cardiovascular modulation. The features were extracted from simultaneously recorded audio and video of remote interviews using task-specific and foundation models. Averages, standard deviations, and hidden Markov model-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination.ResultsStatistically significant feature differences were found between controls and subjects with mental health conditions. Correlations were found between features and self-rated depression and anxiety scores. Visual heart rate dynamics achieved the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities achieved AUROCs of 0.72-0.82. Features from task-specific models outperformed features from foundation models.ConclusionMultimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status.SignificanceThe proposed multimodal approach has the potential to facilitate objective, remote, and low-cost assessment for low-burden automated mental health services.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns

Jiang,

Seyedi,

Griner

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Objective automated digital assessment tools and corresponding digital biomarkers have been widely developed to aid clinicians in addressing the access, subjectivity, and bias challenges [15]. Those tools have been evaluated in various types of mental health disorders, using single data modality [16][17][18] or multiple modalities [19][20][21], in both lab-controlled [22] and remotely collected datasets [23,24].…”

Section: Introductionmentioning

confidence: 99%

Evaluating and mitigating unfairness in multimodal remote mental health assessments

Jiang,

Seyedi,

Griner

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Research on automated mental health assessment tools has been growing in recent years, often aiming to address the subjectivity and bias that existed in the current clinical practice of the psychiatric evaluation process. Despite the substantial health and economic ramifications, the potential unfairness of those automated tools was understudied and required more attention.In this work, we systematically evaluated the fairness level in a multimodal remote mental health dataset and an assessment system, where we compared the fairness level in race, gender, education level, and age.Demographic parity ratio (DPR)andequalized odds ratio (EOR)of classifiers using different modalities were compared, along with the F1 scores in different demographic groups. Post-training classifier threshold optimization was employed to mitigate the unfairness.No statistically significant unfairness was found in the composition of the dataset. Varying degrees of unfairness were identified among modalities, with no single modality consistently demonstrating better fairness across all demographic variables. Post-training mitigation effectively improved both DPR and EOR metrics at the expense of a decrease in F1 scores.Addressing and mitigating unfairness in these automated tools are essential steps in fostering trust among clinicians, gaining deeper insights into their use cases, and facilitating their appropriate utilization.Author summaryIn this work, we systematically explored and discussed the unfairness reporting and mitigation of automated mental health assessment tools. These tools are becoming increasingly important in mental health practice, especially with the rise of telehealth services and large language model applications. However, they often carry inherent biases. Without proper assessment and mitigation, they potentially lead to unfair treatment of certain demographic groups and significant harm. Proper unfairness reporting and mitigation of these tools is the first step to building trust among clinicians and patients and ensuring appropriate application.Using our previously developed multimodal mental health assessment system, we evaluated the unfairness level of using various types of features of the subjects for mental health assessment, including facial expressions, acoustic features of the voice, emotions expressed through language, general language representations generated by large language models, and cardiovascular patterns detected from the face. We analyzed the system’s fairness across different demographics: race, gender, education level, and age. We found no single modality consistently fair across all demographics. While unfairness mitigation methods improved the fairness level, we found a trade-off between the performance and the fairness level, calling for broader moral discussion and investigation on the topic.

show abstract

Mental Disorders Among Mothers in Contact with the Criminal Justice System: A Scoping Review and Meta-analysis

Sapkota,

Dennison,

Thompson

2024

Community Ment Health J

View full text Add to dashboard Cite

This scoping review critically assessed evidence regarding mental disorders among mothers involved with the criminal justice system (CJS) and provided pooled prevalence rates of mental disorders. In total, 27 studies were included in the review, with 23 studies from the United States of America and 26 focused on incarcerated mothers. The findings supported the evidence on substantial burden of mental disorders, among CJS-involved mothers. Several factors contributing to mental disorders were identified, including history of abuse/incarceration/mental illness, a greater number of pregnancies, child-rearing responsibilities, less contact with children, and poor social support, which were organised using a socioecological model. However, significant gaps in the current evidence base were apparent, including inconsistencies in methodologies and outcomes assessed and a lack of large, longitudinal studies. The study highlights the importance of high-quality longitudinal research to extend knowledge around causal pathways between different risk or protective factors and mental disorders among CJS-involved mothers.

show abstract

Multimodal Assessment of Schizophrenia and Depression Utilizing Video, Acoustic, Locomotor, Electroencephalographic, and Heart Rate Technology: Protocol for an Observational Study

Cited by 15 publications

References 75 publications

Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns

Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns

Evaluating and mitigating unfairness in multimodal remote mental health assessments

Mental Disorders Among Mothers in Contact with the Criminal Justice System: A Scoping Review and Meta-analysis

Contact Info

Product

Resources

About