Language models accurately infer correlations between psychological items and scales from text alone

Abstract

Many behavioural scientists do not agree on core constructs and how they should be measured. Different literatures measure related constructs, but the connections are not always obvious to readers and meta-analysts. Many measures in behavioural science are based on agreement with survey items. Because these items are sentences, computerised language models can make connections between disparate measures and constructs and help researchers regain an overview over the rapidly growing, fragmented literature. Our fine-tuned language model, the SurveyBot3000, accurately predicts the correlations between survey items, the reliability of aggregated measurement scales, and intercorrelations between scales from item positions in semantic vector space. In our pilot study, the out-of-sample accuracy for item correlations was .71, .86 for reliabilities, and .89 for scale correlations. In a preregistered study, we will investigate whether the performance of our model generalises to measures across behavioural science.

SEMbeddings: how to evaluate model misfit before data collection using large-language models

“…First of all, we confirmed that cosine similarities of item embeddings resemble empirical item correlations (Hommel & Arslan, 2024). In our case, the correlation between the two lower triangles of the matrices was .56.…”

“…In essence, words exhibit similarities and co-occurrences that mirror the underlying behaviors, thoughts, and emotions they represent. Such similarities and cooccurrences can be effectively captured by modern LLMs (Hommel & Arslan, 2024;Wulff & Mata, 2023).…”

“…This discrepancy arises from the fact that the cosine similarity matrix and the empirical covariance matrix are not perfectly correlated, nor are they on the exact same scale. However, ongoing advancements in LLM models and embedding tools aim to enhance comparability between cosine similarities of embeddings and empirical correlations (Hommel & Arslan, 2024). New models should be adopted if they overperform the ada-002 model that we adopted here.…”