A Comparison of Two Procedures for Constrained Adaptive Test Construction

Abstract

The relatively new shadow test approach (STA) to computerized adaptive testing (CAT) proposed by Wim van der Linden is a potentially attractive alternative to the weighted deviation algorithm (WDA) implemented at ETS. However, it has not been evaluated under testing conditions representative of current ETS testing programs. Of interest was whether STA would, under typical high‐stakes on‐demand testing situations, produce tests of comparable or better psychometric quality as those produced by the current weighted deviation algorithm. Based on simulated data, we found that the STA performed as well or slightly better than the WDA on two of the three commonly accepted testing objectives: measurement and content. The WDA appeared to perform slightly better than the STA when the issue is security or item exposure control. The paper provides a review of the rationale that led to the specific testing objectives employed, an outline of the test construction steps common to the two procedures investigated, and a description of the specific models and algorithms employed by both procedures. Detailed description of the simulation study conducted and the results obtained from both of the procedures for one of the pools are also provided. Results are summarized and further research needs in particular areas discussed.

A Comparison of Four Item-Selection Methods for Severely Constrained CATs

“…The finding that the STA worked better than three heuristic methods not only replicates that in van der Linden (2005) in which the STA has been found to outperform the WDM method but also expands to other heuristic methods. This study, however, seemed to suggest this result came at the cost of the STA making the poorest use of items in comparison with the heuristic methods.…”

“…The necessity of adjusting the weights in implementing the WDM method has been discussed in a few studies (e.g., Cheng & Chang, 2009;van der Linden, 2005). Our study further underscores the critical role of the choice of and the adjustment of weights in the performance of all three heuristic approaches, in particular, with respect to constraint management.…”

“…In most cases, it can be a time-consuming process to find the best weights in order for the tests to achieve the best trade-off among constraints as well as the best trade-off between measurement quality and constraint management. Several studies (Cheng & Chang, 2009;Robin et al, 2005;Shin, Chien, & Way, 2012;van der Linden, 2005) have been conducted to compare the performance of some of the methods discussed above. None of the studies, however, has focused on these four methods simultaneously.…”

A Comparison of Item‐Selection Methods for Adaptive Tests with Content Constraints

“…The shadow test method has been used with the same conditional Sympson‐Hetter method, as well as with an extension of it in which a double‐length shadow test was assembled at each step to create a longer list of items from which the actual item for administration was randomly selected (for details, see Veldkamp & van der Linden, submitted). This extension was used in the study reported by Robin et al (2004). In the empirical study with the adaptive version of the LSAT below, both for the WDM and STA a new item‐exposure control method based on probabilistic item ineligibility constraints was used.…”

“…In addition, we evaluate their performances empirically for an adaptive test from a former item pool of the Law School Admissions Test (LSAT). For an empirical case study with an adaptive test in use at Educational Testing Service (ETS), see Robin, van der Linden, Eignor, Steffen, and Stocking (2004). The differences between this case study and the empirical evaluation in this article are explained below.…”