Do Models of Mental Health Based on Social Media Data Generalize?

Abstract

Proxy-based methods for annotating mental health status in social media have grown popular in computational research due to their ability to gather large training samples. However, an emerging body of literature has raised new concerns regarding the validity of these types of methods for use in clinical applications. To further understand the robustness of distantly supervised mental health models, we explore the generalization ability of machine learning classifiers trained to detect depression in individuals across multiple social media platforms. Our experiments not only reveal that substantial loss occurs when transferring between platforms, but also that there exist several unreliable confounding factors that may enable researchers to overestimate classification performance. Based on these results, we enumerate recommendations for future mental health dataset construction.

Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access

“…The clinically-annotated datasets that do exist are either proprietary or do not provide a clear mechanism for inquiring about availability. The dearth of large, shareable datasets based on actual clinical diagnoses and medical ground truth is problematic given recent research that calls into question the validity of proxy-based mental health annotations (Ernala et al, 2019;Harrigian et al, 2020). By leveraging privacypreserving technology (e.g.…”

“…Mental Health Models. We create mental health models for these datasets based on recent work (Harrigian et al, 2020a;. Following standard pre-processing procedures, we filter numeric values, username mentions, retweets and urls from raw tweet text.…”

“…For model features, we considered TF-IDF vector representations, mean-pooled 200 dimensional Twitter GloVe embeddings , Linguistic Inquiry Word Count (LIWC) representations , and features based on topic distributions learned via LDA. We train ℓ 2 -regularized logistic regression models on both datasets and follow hyper-parameter tuning procedures from Harrigian et al (2020a); .…”

“…Others have used qualitative studies to analyze behaviors and performance of machine learning models in general . Previous work has analyzed representative sentences (Ettinger, 2020), hashtags (Sykora et al, 2020), performed a thematic analysis by using the Linguistic Inquiry and Word Count dictionary or trained topic models (Harrigian et al, 2020a;.…”

“…We base our analysis on datasets from previous work using Twitter. We train simple text-based models based on previous work on these datasets (Harrigian et al, 2020a;. We use a labeled topic model to characterize what content indicates depression and how this content varies by demographic group.…”