Optimal sequential designs for on-line item estimation

Abstract

branch-and-bound, computerized adaptive test, exactn-pointD-optimal, integer programming, item response theory, measurement errors model, on-line testing, sequential design,

Online Calibration Via Variable Length Computerized Adaptive Testing

“…Thus, in Theorem 1 below, it is shown that the asymptotic normality ofβ n requires the measurement error goes to zero faster than the strong consistency ofβ n does. As mentioned before, this assumption is satisfied with either the repeated-sampling method or variable length CAT (Jones &Jin, 1994 andYing, 2004). …”

“…Note that the precision of the estimate of the latent trait level can be improved by increasing the test length under the variable length CAT scenario. Using the replicates of a design to reduce measurement error is also very common idea in measurement error models and it is also mentioned in Stefanski and Carroll (1985) (see also Jones & Jin, 1994). Please see also Li (2002) and Li and Hsiao (2004) for examples in other applications.…”

“…It is shown in the Appendix that for a 2-PL model with adaptive designs following the D-optimal design scheme of Kalish and Rosenberger (1978), the assumptions of Theorems 1 and 2 of Chang (2006) are satisfied, so that we can apply his results to the current online calibration problem. This reparametrization scheme was also used in Jones and Jin (1994); and under such a reparametrized model and assumption of independence of designs, they applied the results of Stefanski and Carroll (1985) to the online calibration problem. Although their numerical results did not show serious flaws, they still mentioned in their paper that under their online calibration setup, the independence assumption of the latent trait levels of examinees is inadequate, since sequential designs in online calibration lack this property.…”

“…In this paper, we select the examinees based on the D-optimal design, which is similar to Jones and Jin (1994), and then use a sequential confidence set estimation method to control the accuracy of item parameters estimation. That is, once the design scheme is fixed, we will keep sampling according to the design scheme until the criterion for constructing the "fixed size" confidence set of the parameters of interest is fulfilled.…”

“…Thus, the assumption of independent observations is no longer valid in this situation. Jones and Jin (1994) applied the measurement error model method to online calibration problems based on Stefanski and Carroll (1985), which relies on the assumption of independence in designs that cannot be fulfilled if an adaptive design scheme is adopted in the calibration procedure. Actually, this flaw was also pointed out in Jones and Jin (1994) (p. 66).…”