Natural language analyzed with AI-based transformers predict traditional subjective well-being measures approaching the theoretical upper limits in accuracy

Abstract

We show that using a recent break-through in artificial intelligence –transformers–, psychological assessments from text-responses can approach theoretical upper limits in accuracy, converging with standard psychological rating scales. Text-responses use people's primary form of communication –natural language– and have been suggested as a more ecologically-valid response format than closed-ended rating scales that dominate social science. However, previous language analysis techniques left a gap between how accurately they converged with standard rating scales and how well ratings scales converge with themselves – a theoretical upper-limit in accuracy. Most recently, AI-based language analysis has gone through a transformation as nearly all of its applications, from Web search to personalized assistants (e.g., Alexa and Siri), have shown unprecedented improvement by using transformers. We evaluate transformers for estimating psychological well-being from questionnaire text- and descriptive word-responses, and find accuracies converging with rating scales that approach the theoretical upper limits (Pearson r = 0.85, p < 0.001, N = 608; in line with most metrics of rating scale reliability). These findings suggest an avenue for modernizing the ubiquitous questionnaire and ultimately opening doors to a greater understanding of the human condition.

Modern Views of Machine Learning for Precision Psychiatry

“…Furthermore, specialized language representations that were trained on mental health specific conversations and are made publicly available [216], have been shown to improve performance compared to non-specific representations. Finally, the use of transformer embedding can be applied to pre-identified sections in language which correspond to responses to standard clinical assessments such as the subjective well-being scales [217], supporting the high-accuracy prediction of the standard survey scale responses without directly running the survey.…”

Three families of automated text analysis

“…Many state-of-the-art algorithms for solving natural language processing tasks (e.g., machine translation, coreference resolution, word sense disambiguation) rely on vector representations acquired via BERT or a close variant. Kjell et al (2022) even find that BERT-based representations of open-ended survey responses measure underlying psychological constructs at least as reliably as traditional, closed-form survey measures.…”

Modern views of machine learning for precision psychiatry

“…Finally, an ML technique known as transformers can assess text responses via NLP and predict traditional subjective well-being measures approaching the theoretical upper limits in accuracy. 224 …”