Optimum Design in Item Response Theory: Test Assembly and Item Calibration

Abstract

The idea of optimizing experimental design to give estimators maximal efficiency has been around in the statistical literature for several decades, but its applicability to sampling problems in item response theory (IRT) has not been widely noticed. It is the purpose of this paper to show how optimum design principles C&ll be used to improve item &lid examinee sampling in IRT-based test assembly and item calibration. For both applications a result based on the maximin principle is given. The maximin principle fits these applications naturally, because IRT models are nonlinear and involve criteria of optimality that are dependent on the unknown parameters. Optimum Design in IRT: Applications to Test Assembly and Item CalibrationThe main topics addressed in statistics are parameter estimation and hypothesis testing. For both topics the literature has produced fruitful methods of finding estimators and test statistics as well as important criteria to evaluate their performances. Most statistical theory is based on the assumption of simple random sampling, and it seems safe to assert that the majority of the statisticians have hardly any interest in sampling beyond this assumption. Two exceptions to this practice are known, though. One is the interest in more complicated sampling procedures than simple random sampling, notably in the domain of survey research (Kalton, 1983; Sarndal, Swensson, and Wretman, 1992). Another exception began with the pioneering work on optimum design of statistical experiments by R. A. Fisher, who, ironically, can also be considered the founder of mainstream statistics. Fisher's work originated in the domain of linear models. Linear models have the important aspect that they focus on the outcomes of statistical experiments in which the statistician may have control over some of the variables, and hence is able to design an experiment in which sampling with respect to estimation of the parameters is optimal. An example is the experiment underlying the estimation of the parameters in a bivariate linear regression model in which the predictor is a fixed variable and the statistician can select the levels of this predictor. Likewise, if the predictor is a random variable, the statistician may have control of the probabilities with which the levels of the lUniversity of Twente,

A Model for Optimal Constrained Adaptive Testing

“…A large variety of conventional test assembly problems have been shown to lend themselves to modeling as an LP problem with 0-1 decision variables. Some relevant references are: Adema (1992aAdema ( , 1992b, Adema, Boekkooi-Timminga, and van der Linden (1991), Adema and van der Linden (1989), , Armstrong, Jones, and Wu (1992), Boekkooi-Timminga (1987, Theunissen (1985Theunissen ( , 1986, Adema (1995, 1996), van der Linden (1994; in press), van der Linden and Boekkooi- , and van der Linden and .…”

Creating IRT-Based Parallel Test Forms Using the Genetic Algorithm Method

“…Many effective methods have been proposed to automatically select items for parallel test forms (e.g., Yen, 1981;Theunissen, 1985;Theunissen, 1986;van der Linden, 1987van der Linden, , 1994van der Linden, , 1998Ackerman, 1989;Boekkooi-Timminga, 1989;van der Linden & Boekkooi-Timminga, 1989;Adema, 1990;Boekkooi-Timminga, 1990;Adema, 1992;Swanson & Stocking, 1993;Wang & Ackerman, 1997;Sanders & Verschoor, 1998;van den Linden & Reese, 1998;Wightman, 1998;Sun & Chen, 1999;and Sun, 2001). The item selection problem can be formulated as a kind of linear programming problem.…”

“…Because the item selection problem is a combinatory optimization problem, the number of combinations increases exponentially with the number of items in the item bank. For this reason, designers typically use weak methods (heuristic algorithms) such as linear programming (LP) techniques that are capable only of finding "good" but not "optimal" solutions (BoekkooiTimminga, 1987;Baker, Cohen, & Barmish, 1988;van der Linden & BoekkooiTimminga, 1989;Swanson & Stocking, 1993;Wang & Ackerman, 1997 (2) Linden, 1998). In linear programming techniques, items are selected to optimize objectives under certain constraints.…”

“…In linear programming techniques, items are selected to optimize objectives under certain constraints. Good solutions can be produced by a variety of other heuristic methods as well-methods such as the branch-and-bound method (Adema, 1989), the revised simplex method (Adema, 1990), the weighted deviation model (Swanson & Stocking, 1993), the network-flow algorithm (Armstrong & Jones, 1992;Armstrong, Jones, & Rutgers, 1996), the 0-1 linear programming (LP) techniques (Adema, 1992;Boekkooi-Timminga, 1990;Theunissen, 1986;van der Linden, 1994van der Linden, , 1998, the neural network technique (Sun & Chen, 1999), and the greedy approach (Sun, 2001).…”

A Strategy for Optimizing Item‐Pool Management

“…Our method of item‐pool design is basically a nonstatistical extension of optimal design methods as they have been developed in statistics (Silvey, 1980) and used, for example, to optimize item calibration and ability estimation problems in IRT (Berger, 1997; van der Linden, 1994). A key notion in optimal design theory is that of a design space.…”