Using Content Coding and Automatic Item Generation to Improve Test Security

Abstract

Automatic item generation (AIG) is the process of using models to generate items using computer technology. AIG is a scalable content development method because it relies on the item model as the unit of analysis which means that it is more efficient and economical compared to traditional item development. But to use the generated items effectively, they must be managed properly. Managing a bank that could include millions of items results in problems related to identifying, organizing, and securing the content. As a result, the challenges inherent to managing item models and generated items warrant a shift in banking methodology where the management task must be accomplished using content coding at the model level. The purpose of our paper is to describe and illustrate methods that use content coding to organize and secure generated items in a bank.

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

“…Although they are feasible for lower-stakes exams, these approaches are less appealing for large-scale exams . The limitations of this procedure are even more pronounced when large number of items are needed to build item banks-pools of items containing information about their content and psychometric details-or parallel test parcels, where each item is treated as an isolated entity (Gierl et al, 2022a;Gierl & Lai, 2013a). Since each item is designed, reviewed and formatted individually, unanticipated psychometric results may arise due to spurious elements that were missed throughout the…”

“…Item Generation: with example from postoperative fever (Gierl, Lai & Turner, 2012). this process, the cognitive and content specific informations needed to solve the problem are organized into a coherent structure that contains the content relationships and information sources necessary to formulate a medical diagnosis in a specific content area (Gierl et al, 2022a;Lai et al, 2016). These models list cognitive and content-specific information, characterize higher-order versus lower-order forms of thinking to solve problems, and provide a theoretical understanding of test performance Leighton & Gierl, 2011).…”

“…Once the cognitive models are determined, the stimulus features will be determined in accordance with these models (Yang et al, 2022). In the second stage, an item model is designed to outline where the content from the cognitive model should be placed in order to generate new items (Gierl et al, 2022a). This model will specify which parts of the item can be manipulated, including the context, content, and the question that examinees are required to answer .…”

“…This model will specify which parts of the item can be manipulated, including the context, content, and the question that examinees are required to answer . For a typical MCQ, it includes the stem (part of the item model that compiles the information needed to solve the problem), response options (all answers, including distractors and correct options), the lead-in-question and other ancillary information (Gierl et al, 2022a). Finally, in the third stage, algorithms integrate the content from the cognitive model outlined in step 1 into the item model developed in step 2 (Gierl et al, 2022a).…”

“…For a typical MCQ, it includes the stem (part of the item model that compiles the information needed to solve the problem), response options (all answers, including distractors and correct options), the lead-in-question and other ancillary information (Gierl et al, 2022a). Finally, in the third stage, algorithms integrate the content from the cognitive model outlined in step 1 into the item model developed in step 2 (Gierl et al, 2022a). Computer algorithms are used to conduct this process because the assembly often involves challenging combinatorial operations (Gierl et al, 2022a).…”