Nemsi

Suendermann-Oeft, David; Robinson, Amanda; Cornish, Andrew; Habberstad, Doug; Pautler, David; Schnelle-Walka, Dirk; Haller, Franziska; Liscombe, Jackson; Neumann, Michael; Merrill, Mike A.; Roesler, Oliver; Geffarth, Renko

doi:10.1145/3308532.3329415

Cited by 10 publications

(4 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Gratch et al [ 228 ] conducted semi-structured interviews with individual participants with both neutral questions and those related to depression or PTSD events, which the authors recorded using a camera and close-talking microphone. In contrast, NEMSI (NEurological and Mental health Screening Instrument) [ 229 ] was proposed as a cloud-based system that automates data capture and the subsequent audio-visual processing for feature extraction and visualization. Before commencing the interviews, researchers [ 228 , 230 ] ensured that participants signed consent forms to collect highly identifiable recording data and share their data for research purposes.…”

Section: Resultsmentioning

confidence: 99%

Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

Khoo,

Lim,

Chong

et al. 2024

Sensors

View full text Add to dashboard Cite

As mental health (MH) disorders become increasingly prevalent, their multifaceted symptoms and comorbidities with other conditions introduce complexity to diagnosis, posing a risk of underdiagnosis. While machine learning (ML) has been explored to mitigate these challenges, we hypothesized that multiple data modalities support more comprehensive detection and that non-intrusive collection approaches better capture natural behaviors. To understand the current trends, we systematically reviewed 184 studies to assess feature extraction, feature fusion, and ML methodologies applied to detect MH disorders from passively sensed multimodal data, including audio and video recordings, social media, smartphones, and wearable devices. Our findings revealed varying correlations of modality-specific features in individualized contexts, potentially influenced by demographics and personalities. We also observed the growing adoption of neural network architectures for model-level fusion and as ML algorithms, which have demonstrated promising efficacy in handling high-dimensional features while modeling within and cross-modality relationships. This work provides future researchers with a clear taxonomy of methodological approaches to multimodal detection of MH disorders to inspire future methodological advancements. The comprehensive analysis also guides and supports future researchers in making informed decisions to select an optimal data source that aligns with specific use cases based on the MH disorder of interest.

show abstract

Section: Resultsmentioning

confidence: 99%

Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

Khoo,

Lim,

Chong

et al. 2024

Sensors

View full text Add to dashboard Cite

show abstract

“…Before participants start their conversation with the virtual agent, Tina—implemented via a scalable, cloud-based MDS to conduct automated structured interactions (Suendermann-Oeft et al, 2019 ; Ramanarayanan et al, 2020 )—tests of the speaker, microphone, and camera need to be passed to ensure that the participants' devices are correctly configured so that the collected data has sufficient quality. Once all device tests pass, Tina guides participants through an interactive interview.…”

Section: Methodsmentioning

confidence: 99%

“…For the present study, the Modality service, a cloud-based MDS (Suendermann-Oeft et al, 2019 ; Ramanarayanan et al, 2020 ) was used to conduct automated, structured interviews with participants. Neumann et al ( 2020 ) recently demonstrated the utility of the Modality MDS in differentiating people with mild, moderate and severe depression, and similar studies have also been conducted in ALS (Neumann et al, 2021 ), Parkinson's disease (Kothare et al, 2022 ), schizophrenia (Richter et al, 2022 ), and autism (Kothare et al, 2021 ).…”

Section: Introductionmentioning

confidence: 99%

A multimodal dialog approach to mental state characterization in clinically depressed, anxious, and suicidal populations

Cohen,

Richter,

Neumann

et al. 2023

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

BackgroundThe rise of depression, anxiety, and suicide rates has led to increased demand for telemedicine-based mental health screening and remote patient monitoring (RPM) solutions to alleviate the burden on, and enhance the efficiency of, mental health practitioners. Multimodal dialog systems (MDS) that conduct on-demand, structured interviews offer a scalable and cost-effective solution to address this need.ObjectiveThis study evaluates the feasibility of a cloud based MDS agent, Tina, for mental state characterization in participants with depression, anxiety, and suicide risk.MethodSixty-eight participants were recruited through an online health registry and completed 73 sessions, with 15 (20.6%), 21 (28.8%), and 26 (35.6%) sessions screening positive for depression, anxiety, and suicide risk, respectively using conventional screening instruments. Participants then interacted with Tina as they completed a structured interview designed to elicit calibrated, open-ended responses regarding the participants' feelings and emotional state. Simultaneously, the platform streamed their speech and video recordings in real-time to a HIPAA-compliant cloud server, to compute speech, language, and facial movement-based biomarkers. After their sessions, participants completed user experience surveys. Machine learning models were developed using extracted features and evaluated with the area under the receiver operating characteristic curve (AUC).ResultsFor both depression and suicide risk, affected individuals tended to have a higher percent pause time, while those positive for anxiety showed reduced lip movement relative to healthy controls. In terms of single-modality classification models, speech features performed best for depression (AUC = 0.64; 95% CI = 0.51–0.78), facial features for anxiety (AUC = 0.57; 95% CI = 0.43–0.71), and text features for suicide risk (AUC = 0.65; 95% CI = 0.52–0.78). Best overall performance was achieved by decision fusion of all models in identifying suicide risk (AUC = 0.76; 95% CI = 0.65–0.87). Participants reported the experience comfortable and shared their feelings.ConclusionMDS is a feasible, useful, effective, and interpretable solution for RPM in real-world clinical depression, anxiety, and suicidal populations. Facial information is more informative for anxiety classification, while speech and language are more discriminative of depression and suicidality markers. In general, combining speech, language, and facial information improved model performance on all classification tasks.

show abstract

“…The Modality service, a cloud-based multimodal dialog system [36, 37, 38], was used to collect video recordings from participants, who engaged in a structured conversation with Tina, a virtual dialog agent (see Figure 2 for a schematic illustration of the dialog platform). To ensure data privacy and protection of personal health information (PHI), the Modality service is fully compliant with the Health Insurance Portability and Accountability Act (HIPAA) and the General Data Protection Regulation (GDPR; European Union).…”

Section: Multimodal Dialog Systemmentioning

confidence: 99%

Multimodal Speech Biomarkers for Remote Monitoring of ALS Disease Progression

Neumann,

Kothare,

Ramanarayanan

2024

Preprint

View full text Add to dashboard Cite

Amyotrophic lateral sclerosis (ALS) is a progressive neurodegenerative disease that severely impacts affected persons’ speech and motor functions, yet early detection and tracking of disease progression remain challenging. The current gold standard for monitoring ALS progression, the ALS functional rating scale - revised (ALSFRS-R), is based on subjective ratings of symptom severity, and may not capture subtle but clinically meaningful changes due to a lack of granularity. Multimodal speech measures which can be automatically collected from patients in a remote fashion allow us to bridge this gap because they are continuous-valued and therefore, potentially more granular at capturing disease progression. Here we investigate the responsiveness and sensitivity of multimodal speech measures in persons with ALS (pALS) collected via a remote patient monitoring platform in an effort to quantify how long it takes to detect a clinically-meaningful change associated with disease progression. We recorded audio and video from 278 participants and automatically extracted multimodal speech biomarkers (acoustic, orofacial, linguistic) from the data. We find that the timing alignment of pALS speech relative to a canonical elicitation of the same prompt and the number of words used to describe a picture are the most responsive measures at detecting such change in both pALS with bulbar (n= 36) and non-bulbar onset (n= 107). Interestingly, the responsiveness of these measures is stable even at small sample sizes. We further found that certain speech measures are sensitive enough to track bulbar decline even when there is no patient-reported clinical change, i.e. the ALSFRS-R speech score remains unchanged at 3 out of a total possible score of 4. The findings of this study have the potential to facilitate improved, accelerated and cost-effective clinical trials and care.

show abstract

Nemsi

Cited by 10 publications

References 7 publications

Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

A multimodal dialog approach to mental state characterization in clinically depressed, anxious, and suicidal populations

Multimodal Speech Biomarkers for Remote Monitoring of ALS Disease Progression

Contact Info

Product

Resources

About