Computerized adaptive testing for testlet‐based innovative items

Abstract

Increasing use of innovative items in operational assessments has shedded new light on the polytomous testlet models. In this study, we examine performance of several scoring models when polytomous items exhibit random testlet effects. Four models are considered for investigation: the partial credit model (PCM), testlet-as-a-polytomousitem model (TPIM), random-effect testlet model (RTM), and fixed-effect testlet model (FTM). The performance of the models was evaluated in two adaptive testings where testlets have nonzero random effects. The outcomes of the study suggest that, despite the manifest random testlet effects, PCM, FTM, and RTM perform comparably in trait recovery and examinee classification. The overall accuracy of PCM and FTM in trait inference was comparable to that of RTM. TPIM consistently underestimated population variance and led to significant overestimation of measurement precision, showing limited utility for operational use. The results of the study provide practical implications for using the polytomous testlet scoring models.

Location-Matching Adaptive Testing for Polytomous Technology-Enhanced Items

“…The b -matched selection is less greedy than the information-based selection and tends to make more exhaustive use of item pools. One issue with applying this criterion to TEIs is that TEIs are typically scored polytomously ( Betts, Muntean, Kim, & Kao, 2021 ; Jiao et al, 2012 ; Kang, Han, Betts, & Muntean, 2022 ), and there is no specific index that can characterize the location of an item. In this study, we introduce location indices that describe the location of a polytomous item on the ability continuum and investigate probable testing outcomes of the location-matched item selection through Monte Carlo simulation.…”

“…5.Previous studies on polytomous CAT commonly considered 100 or fewer items with four response categories (e.g., Gorin et al, 2005; Lee & Dodd, 2012; Leroux et al, 2019; Pastor et al, 2002; Penfield, 2006). This study assumes a moderate size of 200 items with greater score multiplicity to mimic operational TEIs (e.g., Kang, Han, Betts, & Muntean, 2022; Kang, Han, Kim, & Kao, 2022). …”

Location-Matching Adaptive Testing for Polytomous Technology-Enhanced Items

“…The b-matched selection is less greedy than the information-based selection and tends to make more exhaustive use of item pools. One issue with applying this criterion to TEIs is that TEIs are typically scored polytomously (Betts, Muntean, Kim, & Kao, 2021;Kang, Han, Betts, & Muntean, 2022;Jiao, Liu, Hainie, Woo, & Gorham, 2012), and there is no specific index that can represent the location of an item. In this study, we introduce location indices that 1 Copious research exists in the measurement literature that imposes heuristic nonstatistical constraints in item selection (e.g., Chang & Ying, 1999;Cheng & Chang, 2009;Kingsbury & Zara, 1989;Sympson & Hetter, 1985;van der Linden, 2000).…”