Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development.

Abstract

Measurement is at the heart of scientific research. As many-perhaps most-psychological constructs cannot be directly observed, there is a steady demand for reliable self-report scales to assess latent constructs. However, scale development is a tedious process that requires researchers to produce good items in large quantities. In this tutorial, we introduce, explain, and apply the Psychometric Item Generator (PIG), an opensource, free-to-use, self-sufficient natural language processing algorithm that produces large-scale, humanlike, customized text output within a few mouse clicks. The PIG is based on the GPT-2, a powerful generative language model, and runs on Google Colaboratory-an interactive virtual notebook environment that executes code on state-of-the-art virtual machines at no cost. Across two demonstrations and a preregistered fivepronged empirical validation with two Canadian samples (N Sample 1 = 501, N Sample 2 = 773), we show that the PIG is equally well-suited to generate large pools of face-valid items for novel constructs (i.e., wanderlust) and create parsimonious short scales of existing constructs (i.e., Big Five personality traits) that yield strong performances when tested in the wild and benchmarked against current gold standards for assessment. The PIG does not require any prior coding skills or access to computational resources and can easily be tailored to any desired context by simply switching out short linguistic prompts in a single line of code. In short, we present an effective, novel machine learning solution to an old psychological challenge. As such, the PIG will not require you to learn a new language-but instead, speak yours.

The Potential of Machine Learning Methods in Psychological Assessment and Test Construction

“…Because they are trained on vast amounts of data, they are able to use contextual information to generate text that is more human-like than that of previous statistical language models (Demszky et al, 2023). For example, Götz et al (2023) recently introduced the Psychometric Item Generator, a tool for generating questionnaire items that are based on GPT-2 (Radford et al, 2019). Using this tool, they were able to create a new, shorter version of a Big Five questionnaire that showed similar psychometric properties to the original, though human judgments of the generated items were necessary to pre-select items for this new questionnaire version.…”

Can Generative AI improve social science?

“…Though silicon samples will not soon displace survey research with human respondents, they may still be very useful for pretesting surveys before they are dispatched (at considerable cost) to large groups of human respondents, or imputing missing data ( 11 , 17 ). Some argue Generative AI is also a useful tool for creating survey questions, or designing multi-item scales to measure abstract social concepts ( 18 ).…”

The Structure of Self-Related Core Beliefs

“…Today, natural language processing approaches are used in many everyday life applications such as translation programs, autocorrect functions, dialogue systems, part-ofspeech tagging, search for plagiarism, text classification, text extraction, or text generation, such as ChatGPT (OpenAI, 2023;Young et al, 2018). Also, there are first applications in psychology, including generation of items for psychological tests (Götz et al, 2022); automatic analysis of free text responses (Gratz et al, 2022); investigating semantic similarities in situation descriptors (Parrigon et al, 2017), or producing variance-covariance matrices for personality adjectives (Cutler & Condon, 2023), giving rise to a new field of AI psychology.…”