A comparison of panel designs with routing methods in the multistage test with the partial credit model

Abstract

In this study, we compared panel designs applied with various routing methods in the multistage test (MST) based on the partial credit model in the context of classification testing. Simulations were performed to compare three routing methods and four panel structures. Conditions of two test lengths and three passing rates were also included. The results showed that, regardless of the routing method used, the same panel structure performed similarly in terms of the precision of the classification decision with the same test length condition. The longer test length produced higher accuracy, whereas the 50 % passing rate yielded the lowest accuracy. Finally, all MST conditions performed well in terms of test security.

The Development of MST Test Information for the Prediction of Test Performances

“…For the three routing methods of a defined population interval (DPI; Luecht et al, 2006), proximity, and number-correct scores (NC), the average proportions of examinees following primary pathways were 81.74%, 81.6%, and 71.15% for the 1-3-3 structure and 90.13%, 79.8%, and 81.34% for the 1-2-2 MST design. Furthermore, Kim, Chung, Park and Dodd (2013) compared MST panel designs using the partial credit model and reported that the use of primary pathways was on average 73% and 78% for approximate maximum information (AMI) and DPI routing methods, respectively.…”

Ability Estimation with Polytomous Items in Computerized Multistage Tests

“…Various past studies on routing methods generally apply Approximate Maximum Information (AMI), Defined Population Intervals (DPI), and convergent and random routing methods (Kim et al, 2010;Zenisky, 2004). Routing methods that are based on IRT are other frequently used kinds.…”

“…In a 1-3 panel design, the difficulty levels of the modules are usually determined as easy, medium, and complicated in the 2nd stage. 1-2, 1-2-2, 1-3, 1-3-3, 1-4, and 1-4-4 panel designs, which are preferred in the literature, were used in this study (Kim et al, 2010;Oztürk, 2019;Sarı & Raborn, 2018).…”

Contrasting multistage and computer-based testing: score accuracy and aberrant responding

“…Therefore, MST aims to reduce errors in estimating item parameters and ability levels and also the length of the test ( Glas, 1988 ; Zheng and Chang, 2015 ). Multistage testing has recently undergone increased adoption as an alternative to both the classical linear test (CLT) and CAT ( Kim et al, 2013 ). A multistage test is designed by selecting items from a pool that was calibrated before the test was administered previously, which benefits both developers and examinees; MST gives test developers more control over content balancing, item difficulty’s distribution, the quality of the test structure, dependencies among the items, and the distribution of non-statistical properties of the items such as the cognitive level.…”

“…The latter engages maximum likelihood (MLE) or expected a posteriori (EAP) estimation procedure [10]. Based on performance during the first stage, a participant is” routed” to a module that is closer to their ability level in the second stage of a panel design ( Kim et al, 2013 ). This procedure is followed in several stages until all modules are administered within a design.…”