A Feasibility Study of On‐the‐fly Item Generation in Adaptive Testing

Abstract

The goal of this study was to assess the feasibility of an approach to adaptive testing based on item models. A simulation study was designed to explore the affects of item modeling on score precision and bias, and two experimental tests were administered -an experimental, on-the-fly, adaptive quantitative-reasoning test as well as a linear test. Results of the simulation study showed that under different levels of isomorphicity, there was no bias, but precision of measurement was eroded, especially in the middle range of the true-score scale. However, the comparison of adaptive test scores with operational Graduate Record Examinations (GRE) test scores matched the test-retest correlation observed under operational conditions. Analyses of item functioning on linear forms suggested a high level of isomorphicity across items within models. The current study provides a promising first step toward significant cost and theoretical improvement in test creation methodology for educational assessment.

Defining Mathematics Competency in the Service of Cognitively Based Assessment for Grades 6 Through 8

“…Second, the approach may save on pretesting costs. If the psychometric parameters of model-generated instances may be predicted successfully in advance, it may not be necessary to calibrate each instance individually (e.g., Bejar, 1993Bejar, , 1996Bejar et al, 2002;Bejar & Yocom, 1991;Embretson, 1999). Several researchers have explored the extent to which item models generate isomorphs, or instances with highly similar psychometric parameters (Meisner, Luecht, & Reckase, 1993;Sinharay & Johnson, 2005;Steffen, Graf, Levin, Robin, & Lu, 2006).…”

On the Roles of External Knowledge Representations in Assessment Design

“…These task construction schemas can themselves be expressed in terms of KRs. Hively, Patterson, and Page's (1968) item shells and Haladyna and Shindoll's (1989) item forms represented initial research along these lines, while more recent technology-based task construction frameworks include those of Bejar et al (2003), Gierl, Zhou, and Alves (2008), and .…”

“…When a task model is in digital form and the slots are appropriately filled, the resulting form can serve as input to subsequent processes to create tasks in the forms in which they are needed in implementation and presentation (as in Bejar et al, 2003, Gierl et al, 2008, Hamel, Mislevy, & Winters, 2008, and Hamel & Schank, 2006. Two examples of programs that can facilitate task authoring using the idea of item templates are Mathematics Test Creation Assistant (TCA, Singley & Bennett, 2002) and the Free-Response Authoring, Delivery, and Scoring System (FRADSS, Katz, 1995).…”

Information-based methods for evaluating the semantics of automatically generated test items

“…The outcome of this algorithmic transcription is often referred to as item modeling. [3][4][5][6] The item modeling process requires the identification of elements within the assessment task so that these elements can be used to create large set of items. [1,7] These generated items may or may not be similar to one another and thus the lexical similarity among the generated items is often unknown.…”