The Seven Wonderings of Large Language Models as Psychometric Designers, Refiners, and Analysts

Abstract

The irruption of Large Language Models (LLMs) in our daily lives has opened up an intriguing future for the course of psychometrics. We give a glimpse of that future through our seven “wonderings” of LLMs: a series of wanderings on how current LLMs can instill wonder in researchers and professionals by assisting them in each step of the design, refinement, and analysis of psychometric tools. Using GPT-4 as illustration, we have tried to answer what are the capabilities of LLMs as item designers and format generators, as reviewers and respondents, and as data analysts and results interpreters. We interacted with the LLM applying a systematic prompt scheme, evidencing the peaks and pitfalls of its responses when addressing psychometric tasks. Finally, we provide some thoughts and guidelines about the validity of the uses LLMs responses can offer, and how to study and perform such validation process.