Machine learning of language use on Twitter reveals weak and non-specific predictions

Abstract

Background: Depressed individuals use language differently than healthy controls and it has been proposed that social media posts could therefore be used to identify depression. But much of the evidence behind this claim relies on indirect measures of mental health that are sometimes circular, such as statements of self-diagnosis (“Got an OCD diagnosis today”) on social media or membership in disorder-specific online forums. Relatedly, few studies have tested if these language features are specific to depression versus other aspects of mental health. Methods: We analyzed the Tweets of 1,006 participants who completed questionnaires assessing symptoms of depression and 8 other mental health conditions. Daily Tweets were subjected to textual analysis and the resulting linguistic features were used to train an Elastic Net model on depression severity, using nested cross validation. We then tested performance in a held-out test set (30%), comparing predictions of depression versus 8 other aspects of mental health. Results: The depression trained model had only modest predictive performance when tested out of sample, explaining just 2.5% of variance in depression symptoms (R2 = 0.025). The performance of this model was as-good or superior when used to identify other aspects of mental health: schizotypy (R2 = 0.035), social anxiety (R2 = 0.025), eating disorders (R2 = 0.025), generalized anxiety (R2 = 0.041), above chance for obsessive-compulsive disorder (R2 = 0.011), apathy (R2 = 0.008), but not significant for alcohol abuse (R2 = -0.012).

Optimising the measurement of anxious-depressive, compulsivity and intrusive thought and social withdrawal transdiagnostic symptom dimensions

“…A further dataset was obtained from Kelley et al (2021), n = 1006, with participants recruited through Clickworker https://www.clickworker.com/. The final dataset (n = 2460) was unpublished data obtained from the smartphone app Neureka https://www.neureka.ie/.…”

“…The final dataset (n = 2460) was unpublished data obtained from the smartphone app Neureka https://www.neureka.ie/. The Rouault et al (2018) and Patzelt et al (2019) data were collected before the COVID-19 pandemic, the Kelley et al (2021) and Neureka datasets were obtained during the COVID-19 pandemic.…”

“…Skewed variables, such as the OCIR, AUDIT and EAT scales were transformed using square root transformation. All questionnaires had significant substudy differences, with follow-up Tukey HSD tests indicating that the Kelley et al (2021) dataset was generally significantly higher in symptoms than others, except for social anxiety (Table S1).…”