Optimal Item Calibration for Computerized Achievement Tests

Abstract

Item calibration is a technique to estimate characteristics of questions (called items) for achievement tests. In computerized tests, item calibration is an important tool for maintaining, updating and developing new items for an item bank. To efficiently sample examinees with specific ability levels for this calibration, we use optimal design theory assuming that the probability to answer correctly follows an item response model. Locally optimal unrestricted designs have usually a few design points for ability. In practice, it is hard to sample examinees from a population with these specific ability levels due to unavailability or limited availability of examinees. To counter this problem, we use the concept of optimal restricted designs and show that this concept naturally fits to item calibration. We prove an equivalence theorem needed to verify optimality of a design. Locally optimal restricted designs provide intervals of ability levels for optimal calibration of an item. When assuming a two-parameter logistic model, several scenarios with D-optimal restricted designs are presented for calibration of a single item and simultaneous calibration of several items. These scenarios show that the naive way to sample examinees around unrestricted design points is not optimal. Electronic supplementary materialThe online version of this article (10.1007/s11336-019-09673-6) contains supplementary material, which is available to authorized users.

Optimal Calibration of Items for Multidimensional Achievement Tests

“…The approach used in this research study is a multidimensional generalization of the approach described in Ul Hassan and Miller (2019b) for optimal subpopulation sampling. Suppose the population of examinees participating in the computerized test has a multivariate continuous probability density $$g$$ on $$\Theta$$.…”

“…Theorem 1Ul Hassan and Miller (2019b): Suppose h * is an optimal restricted design in the set of all restricted designs g and L is defined as ( 6) and is differentiable. Then, h * is -optimal if and only if…”

“…Ul Hassan and Miller (2019b) have developed a restricted optimal design approach for item calibration. They included the restriction on availability of examinees in their approach and proved an equivalence theorem to ensure that a design for item calibration is optimal.…”

“…The complexity of the algorithm increases with the increase in the number of items to calibrate. Therefore, Ul Hassan and Miller (2021, 2023) have developed another algorithm for computing restricted optimal design to calibrate all experimental items simultaneously and provided an R package with this algorithm. In this research study, we generalized this algorithm for simultaneously calibration of items in a multidimensional setting.…”

“…The structure of optimal calibration designs have been illustrated by Ul Hassan and Miller (2019b, 2021) for two unidimensional IRT models. Examinees with abilities in certain intervals are allocated to each item.…”