Using automatic item generation to create multiple-choice test items

Abstract

Medical Education 2012: 46: 757–765 Context Many tests of medical knowledge, from the undergraduate level to the level of certification and licensure, contain multiple‐choice items. Although these are efficient in measuring examinees’ knowledge and skills across diverse content areas, multiple‐choice items are time‐consuming and expensive to create. Changes in student assessment brought about by new forms of computer‐based testing have created the demand for large numbers of multiple‐choice items. Our current approaches to item development cannot meet this demand. Methods We present a methodology for developing multiple‐choice items based on automatic item generation (AIG) concepts and procedures. We describe a three‐stage approach to AIG and we illustrate this approach by generating multiple‐choice items for a medical licensure test in the content area of surgery. Results To generate multiple‐choice items, our method requires a three‐stage process. Firstly, a cognitive model is created by content specialists. Secondly, item models are developed using the content from the cognitive model. Thirdly, items are generated from the item models using computer software. Using this methodology, we generated 1248 multiple‐choice items from one item model. Conclusions Automatic item generation is a process that involves using models to generate items using computer technology. With our method, content specialists identify and structure the content for the test items, and computer technology systematically combines the content to generate new test items. By combining these outcomes, items can be generated automatically.

A Model-Based Method for Content Validation of Automatically Generated Test Items

“…The outcomes from the second and third examples can be obtained from the first author. All the generated items were created by medical SMEs using the three-step AIG process that included developing a cognitive model, creating an item model, and generating items (Gierl, Lai, &Turner, 2012). The items in each dataset were generated based on individual cognitive models and the derived item models.…”

“…The first step in the AIG process is to develop cognitive models which highlight both the examinees' knowledge and skills required to solve the item as well as specify the content features in the items. To create the cognitive models, the SMEs are asked to identify and describe the key information that would be used to solve a test item (Gierl, Lai, & Turner, 2012). This cognitive model is used to guide the detailed rendering needed for item generation.…”

Near-peer question writing and teaching programme

“…Current literature involves the creation of complex algorithms or computer software, with low‐validity question banks or workshops that are either descriptive or too technical for peer teachers to reproduce. By having a structured format to question writing and question selection, we tried to negate the common criticism of near‐peer teaching and peer‐assisted teaching: i.e.…”

Medical school 2.0: How we developed a student-generated question bank using small group learning

“…The instructor garnered great educational satisfaction and enjoyment from the session, and was saved from creating new MCQs from scratch. Previous reports describe creating questions using computers (Gierl et al 2012), innovative workshops (Droegemueller et al 2005), and questions from other schools (Freeman et al 2010) to augment an institution's question bank. The curriculum also gained an influx of multiple-choice questions that have undergone multiple steps of review.…”