An Evaluation of Automatic Item Generation: A Case Study of Weak Theory Approach

Abstract

This case study applied the weak theory of Automatic Item Generation (AIG) to generate isomorphic item instances (i.e., unique but psychometrically equivalent items) for a large-scale assessment. Three representative instances were selected from each item template (i.e., model) and pilot-tested. In addition, a new analytical framework, differential child item functioning (DCIF) analysis, based on the existing differential item functioning statistics, was applied to evaluate the psychometric equivalency of item instances within each template. The results showed that, out of 23 templates, nine successfully generated isomorphic instances, five required minor revisions to make them isomorphic, and the remaining templates required major modifications. The results and insights obtained from the AIG template development procedure may help item writers and psychometricians effectively develop and manage the templates that generate isomorphic instances.

A Workflow for Minimizing Errors in Template‐Based Automated Item‐Generation Development

“…Given the confidence built around the template, the rest of the 10 items may not need to be reviewed and can be directly field‐tested to obtain the statistics. Moreover, if the items from a template are proven to be isomorphic, meaning they are not significantly different in psychometric properties, field‐testing them may not be necessary (Bejar et al., 2003; Fu et al., 2022).…”

“…Specifically, if the distractors are randomly generated, the item writers need to check if generated distractors and key options can be duplicated. Moreover, the distractor values should not be too implausible so that the examinees can quickly eliminate them because having an implausible distractor can impact the quality of the item (Fu et al., 2022). On the other hand, the distractor values can be very similar to the key value to make the item harder to guess.…”

“…The practitioners can use key values of the items to select representative quantitative items (Fu et al., 2022) for field‐testing. Different combinations of the numeric values may produce different key values, and the practitioners can select items based on the range of the key values.…”