The interactive reading task: Transformer-based automatic item generation

Abstract

Automatic item generation (AIG) has the potential to greatly expand the number of items for educational assessments, while simultaneously allowing for a more construct-driven approach to item development. However, the traditional item modeling approach in AIG is limited in scope to content areas that are relatively easy to model (such as math problems), and depends on highly skilled content experts to create each model. In this paper we describe the interactive reading task, a transformer-based deep language modeling approach for creating reading comprehension assessments. This approach allows a fully automated process for the creation of source passages together with a wide range of comprehension questions about the passages. The format of the questions allows automatic scoring of responses with high fidelity (e.g., selected response questions). We present the results of a large-scale pilot of the interactive reading task, with hundreds of passages and thousands of questions. These passages were administered as part of the practice test of the Duolingo English Test. Human review of the materials and psychometric analyses of test taker results demonstrate the feasibility of this approach for automatic creation of complex educational assessments.

Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI

“…As an example, Figure 1 describes the automated content generation process with human‐in‐the‐loop used in the Duolingo English Test (Attali et al., 2022). Human experts stay in the loop of construct definition, task design, item generation, and refinement to review item quality, fairness, and bias.…”

Using OpenAI GPT to Generate Reading Comprehension Items

“…Similarly, Attali et al. (2022) used GPT‐3 to create interactive reading passages involving human reviewers. Bezirhan and von Davier (2023) also sought expert opinions to assess the quality of the texts generated with GPT.…”

“…Von Davier (2018) carried out a study examining automatically generated items through an online survey and used experts' opinions for the validation. Similarly, Attali et al (2022) used GPT-3 to create interactive reading passages involving human reviewers. Bezirhan and von Davier (2023) also sought expert opinions to assess the quality of the texts generated with GPT.…”

Future challenges and opportunities in language testing and assessment: Basic questions and principles at the forefront

“…This issue is, for example, illustrated by Shi and Aryadoust’s (2022) systematic review of automated writing evaluation (AWE) systems, which concluded that domain definition inferences are underrepresented in AWE research. Similarly, even the most sophisticated automatic item generation (AIG) systems (e.g., Attali et al, 2022), while making great advances, are currently still trained on more “traditional,” “conventional” language use, and restricted in the kind of tasks (input and questions) they can produce (as well as continuing to require considerable human reviewing). Thus, a challenge for future work on technology in language testing and assessment will be to reflect the progress made in (applied) linguistics regarding the nature of language and language use in society, and to prioritize construct operationalization insights, in order to avoid restricting domain representation through technology-mediated testing.…”