Transformer-Based Deep Neural Language Modeling for Construct-Specific Automatic Item Generation

Abstract

Algorithmic automatic item generation can be used to obtain large quantities of cognitive items in the domains of knowledge and aptitude testing. However, conventional item models used by template-based automatic item generation techniques are not ideal for the creation of items for non-cognitive constructs. Progress in this area has been made recently by employing long short-term memory recurrent neural networks to produce word sequences that syntactically resemble items typically found in personality questionnaires. To date, such items have been produced unconditionally, without the possibility of selectively targeting personality domains. In this article, we offer a brief synopsis on past developments in natural language processing and explain why the automatic generation of construct-specific items has become attainable only due to recent technological progress. We propose that pre-trained causal transformer models can be fine-tuned to achieve this task using implicit parameterization in conjunction with conditional generation. We demonstrate this method in a tutorial-like fashion and finally compare aspects of validity in human- and machine-authored items using empirical data. Our study finds that approximately two-thirds of the automatically generated items show good psychometric properties (factor loadings above .40) and that one-third even have properties equivalent to established and highly curated human-authored items. Our work thus demonstrates the practical use of deep neural networks for non-cognitive automatic item generation.

Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development.

“…Future applications may yield even better results by retraining the GPT-2 on custom-tailored text corpora, such as psychological scale databases. Indeed, first examples of targeted fine-tuning are already producing encouraging initial results and foreshadow the minute task-specific calibration that may be accomplished in the future (Hommel et al, 2021; Howard & Ruder, 2018; von Davier, 2019), while other scholars suggest that the ever-increasing powers of coming generations of generative language models will soon remove the need and incentive for retraining as peak performance can be accomplished without requiring any further fine-tuning (Han et al, 2021; Launay et al, 2021; Yang, 2021).…”

Pseudo Factor Analysis of Language Embedding Similarity Matrices: New Ways to Model Latent Constructs

“…Recent work in psychometrics has capitalized on the availability of transformer models to improve psychological measurement (Demszky et al 2023;Hao et al 2024;Hommel et al 2022). Continuing this line of work, Guenole, Samo, & Sun (2024) introduced the idea of pseudo-discrimination using transformer-based large language models (LLMs).…”

MxML (Exploring the Relationship between Measurement and Machine Learning): Current State of the Field

“…For instance, earlier generation automated item generation (AIG) tools (e.g., Gierl & Haladyna, 2012) required test developers to construct item blueprints with articulated factors from prespecified cognitive models. Large language models (LLMs), part of the ML toolkit, can assist test developers generate hundreds of thousands of items quickly, at scale, and with a large degree of variability (e.g., Belzak et al., 2023; Bezirhan & von Davier, 2023; Hommel et al., 2021; Laverghetta & Licato, 2023; von Davier, 2018). These items can be administered to candidates on large‐scale standardized tests, leading to high‐dimensional datasets with large degrees of sparsity across items of various response formats.…”