A Comparison of Four Item-Selection Methods for Severely Constrained CATs

Abstract

This study compared four item-selection procedures developed for use with severely constrained computerized adaptive tests (CATs). Severely constrained CATs refer to those adaptive tests that seek to meet a complex set of constraints that are often not conclusive to each other (i.e., an item may contribute to the satisfaction of several constraints at the same time). The procedures examined in the study included the weighted deviation model (WDM), the weighted penalty model (WPM), the maximum priority index (MPI), and the shadow test approach (STA). In addition, two modified versions of the MPI procedure were introduced to deal with an edge case condition that results in the item selection procedure becoming dysfunctional during a test. The results suggest that the STA worked best among all candidate methods in terms of measurement accuracy and constraint management. For the other three heuristic approaches, they did not differ significantly in measurement accuracy and constraint management at the lower bound level. However, the WPM method appears to perform considerably better in overall constraint management than either the WDM or MPI method. Limitations and future research directions were also discussed.

Variable neighborhood search heuristics for a test assembly design problem

“…These results apply under varying conditions, such as when the items are grouped according to similar weights, as in Brusco et al (2013), or when the items are randomly grouped. Results for randomly grouped items consider cases in which additional constraints are to be enforced (He et al, 2014;Ishii et al, 2014;Nguyen & Fong, 2013), or cases in which a cell approach (Chen et al, 2012;Chen, 2015) is used to maintain similar characteristics not only among questionnaires, but also among the item composition of each questionnaire.…”

“…These results lead us to conjecture that the proposed procedure could also be used for on-line test construction methods as in van der Linden & Veldkamp (2004) or He et al (2014).…”

“…In He et al (2014), four different item-selection methods are evaluated. These methods take into account multiple particularities, such as side constraints, content specifications, time requirements, or item formats, among others.…”

“…For smaller sized instances, the method provides solutions of the 210 same quality within seconds. These results open the door to their use in online single test construction methods using the shadow test approach (He et al, 2014). The shadow test approach is a online single test construction method in which a complete test is constructed in each CAT decision step and the final item used is selected from the constructed test.…”

Heuristic Constraint Management Methods in Multidimensional Adaptive Testing

“…According to the literature, multiple constraint management methods (CMMs) can account for test specifications during the CAT item selection process. He, Diao, and Hauser (2014) gave a brief overview of the existing CMM and differentiated between two types. The first type of CMM, such as the constrained CAT method (Kingsbury & Zara, 1991) and modified multinomial model (Chen & Ankenman, 2004) can only address mutually exclusive constraints.…”