Automatic item generation: foundations and machine learning-based approaches for assessments

Abstract

This mini review summarizes the current state of knowledge about automatic item generation in the context of educational assessment and discusses key points in the item generation pipeline. Assessment is critical in all learning systems and digitalized assessments have shown significant growth over the last decade. This leads to an urgent need to generate more items in a fast and efficient manner. Continuous improvements in computational power and advancements in methodological approaches, specifically in the field of natural language processing, provide new opportunities as well as new challenges in automatic generation of items for educational assessment. This mini review asserts the need for more work across a wide variety of areas for the scaled implementation of AIG.

AI in medical education: uses of AI in construction type A MCQs

“…Using AI and AI-driven applications, tools, and techniques can decrease the challenge of traditional methods presented as item construction and item stability [ 23 ]. Through AI, it will be easier and more feasible to construct and update items (questions) and form item banks.…”

A comparative study of AI-human-made and human-made test forms for a university TESOL theory course

“…Many items can be generated for a specific topic based on a single cognitive model (Gierl et al, 2012), and the models are standards in measurement theories, allowing the developed tests to serve the assessment purposes of validity, reliability, fairness, and quality. Moreover, AIG is known to make test and assessment development easier by making it quicker to create items, reducing the cost of item creation, helping to continuously and rapidly develop a large pool of items, and tailoring items to fit individual learning needs for better outcomes (Circi et al 2023).…”

“…In educational settings, there has been a growing demand for the rapid generation of assessment items to accommodate continuous testing requirements (Kurdi et al, 2019). This shift has posed challenges to traditional test item creation methods and to the maintenance of test item bank stability (Circi et al, 2023). Finding high-quality test items has consistently proven difficult, with the manual creation of items being timeconsuming and costly (Gehringer, 2004).…”

“…Many teachers encounter difficulties designing quality items for each assessment, often resorting to item reuse across terms (Gehringer, 2004;Wellberg, 2023). However, this practice may lead to issues such as students memorizing answers without engaging with the content and the risk of cheating through item over-exposure (Circi et al, 2023;Gehringer, 2004).…”

The Seven Wonderings of Large Language Models as Psychometric Designers, Refiners, and Analysts

“…In that moment such tools were the specific softwares (e.g., IGOR), which allowed psychometricians to create AIG algorithms (Mortimer et al, 2012). The most common strategy was model-based AIG, which consists of creating an item template with certain modifiable elements (the part that can be automatically generated), based on week or strong cognitive models involved in the response process (Circi et al, 2023). Since this method is hard to apply to the nature of non-cognitive items, AIG has been focused on test formats such as multiple-choice to measure specific knowledge and skills (Gierl & Lai, 2018;Gierl et al, 2008;Pugh et al, 2016), or intelligence (K. Wang & Su, 2015).…”