Natural Language Analyzed with AI-based Transformers Predict Traditional Well-Being Measures Approaching the Theoretical Upper Limits in Accuracy

Abstract

We show that using a recent break-through in artificial intelligence –transformers–, psychological assessments from text-responses can approach theoretical upper limits in accuracy, converging with standard psychological rating scales. Text-responses use people's primary form of communication –natural language– and have been suggested as a more ecologically-valid response format than closed-ended rating scales that dominate social science. However, previous language analysis techniques left a gap between how accurately they converged with standard rating scales and how well ratings scales converge with themselves – a theoretical upper-limit in accuracy. Most recently, AI-based language analysis has gone through a transformation as nearly all of its applications, from Web search to personalized assistants (e.g., Alexa and Siri), have shown unprecedented improvement by using transformers. We evaluate transformers for estimating psychological well-being from questionnaire text- and descriptive word-responses, and find accuracies converging with rating scales that approach the theoretical upper limits (Pearson r = .85, p < .001, N = 608; in line with most metrics of rating scale reliability). These findings suggest an avenue for modernising the ubiquitous questionnaire and ultimately opening doors to a greater understanding of the human condition.

Precise language responses versus easy rating scales—Comparing respondents’ views with clinicians’ belief of the respondent’s views

“…The findings that participants rate free text response to be more precise in communicating mental health compared to rating scales is consistent with recent findings showing that QCLA has a high validity in measuring mental health. For example, Kjell et al [ 2 , 7 ] used natural language processing algorithms (e.g., Latent Semantic Analysis [LSA]; [ 17 ]; BERT, [ 5 ]) to map the text responses to vectors representing participants’ descriptions of their mental health. They then used machine learning (ML, multiple linear regression) to build a model to predict ratings scales for mental health (e.g., PHQ-9 and GAD-7).…”

“…Data of participants that did not complete the study was not saved. Participants with MDD and GAD have been systematically investigating using QCLA in several previous studies (e.g., [ 2 , 7 , 11 ]). A drawback with this choice was that a smaller group of the participants only had a GAD diagnosis (N = 21), whereas most of them had MDD, which is the focus of the current study (N = 145 and sometimes also GAD).…”

“…For example, BERT (Bidirectional Encoder Representations from Transformers; [ 5 ]) is a language model based on deep learning neural networks that use transformers [ 6 ], which has greatly improved the accuracy of NLP models. Recently, BERT was used to predict rating scales with an accuracy ( r = .85) that challenges the theoretical upper limits of test-retest reliability [ 7 ]. This was accomplished using multiple free texts and descriptive word responses regarding harmony in life and satisfaction with life as inputs.…”

“…In the analyses, we focused primarily on comparing the free text and rating scales, where we expected the descriptive and selected word condition to have intermediate rating scores. Once again, the choice of these four formats was based on the fact that we have previously used them in earlier studies (e.g., [ 2 , 7 ]). We then asked the respondents to rate their view of the response format concerning the following 12 dimensions.…”

Precise Language Responses Challenge Easy Rating Scales - Comparing Clinicians’ and Respondents’ Views

“…In the analyses we focused primarily on comparing the free text and rating scales, where we expected the descriptive and selected words condition to have intermediate rating scores. The choice of these four formats was based on the fact that we previously have been using them in earlier studies (e.g., Kjell et al, 2019;Kjell, Sikström, Kjell, Schwartz, 2021). We then asked the respondents to rate their view of the response format on the following 12 dimensions.…”