Comparison Between Dichotomous and Polytomous Scoring of Innovative Items in a Large-Scale Computerized Adaptive Test

Abstract

This study explored the impact of partial credit scoring of one type of innovative items (multiple-response items) in a computerized adaptive version of a large-scale licensure pretest and operational test settings. The impacts of partial credit scoring on the estimation of the ability parameters and classification decisions in operational test settings were explored in one real data analysis and two simulation studies when two different polytomous scoring algorithms, automated polytomous scoring and rater-generated polytomous scoring, were applied. For the real data analyses, the ability estimates from dichotomous and polytomous scoring were highly correlated; the classification consistency between different scoring algorithms was nearly perfect. Information distribution changed slightly in the operational item bank. In the two simulation studies comparing each polytomous scoring with dichotomous scoring, the ability estimates resulting from polytomous scoring had slightly higher measurement precision than those resulting from dichotomous scoring. The practical impact related to classification decision was minor because of the extremely small number of items that could be scored polytomously in this current study.

Development of a Computerized Adaptive Test for Schizotypy Assessment

“…Thus, this artificial dichotomization of the ESQUIZO-Q scores sacrifices information in favor of the CAT administration process. At this point, it is worth mentioning that studies in the field of educational measurement have found that ability estimates resulting from polytomous scoring had slightly higher measurement precision than those resulting from dichotomous scoring [57].…”

Evaluating Different Scoring Methods for Multiple Response Items Providing Partial Credit

“…Early research on MR response type highlighted the potential for evaluating partial knowledge and found positive results (Collet, 1971; Coombs et al, 1956; Cronbach, 1939, 1941; Dressel & Schmid, 1953; Morgan, 1979). Likewise, more recent research explored alternative scoring solutions that focused on assigning raw score values to option response sets (Bauer et al, 2011; Becker & Soni, 2013; Betts, 2013, 2018; Clyne, 2015; Couch et al, 2018; Domnich et al, 2015; Dressel & Schmid, 1953; Eggen & Lampe, 2011; Hohensinn & Kubinger, 2011; Hsu et al, 1984; Jiao et al, 2012; Jorion et al, 2019; Kao & Betts, 2019; Kim et al, 2018; Lorié, 2014; Muckle et al, 2011; Muntean & Betts, 2015, 2016a, 2016b; Muntean et al, 2019). These studies have generally shown positive results for a broad range of partial credit scoring models.…”

Computerized adaptive testing for testlet‐based innovative items

“…We especially consider the models that have practical utility for innovative items. In real assessments, innovative items are often administered in testlets and scored polytomously, allowing partial credits for multiple responses (e.g., choose all that apply, multiple true/false; Betts, Muntean, Kim, & Kao, 2021;Davey, Godwin, & Mittelholtz, 1997;Jiao, Liu, Hainie, Woo, & Gorham, 2012;Jodoin, 2003). While a seemingly workable approach to modelling such items is to apply polytomous scoring or use the Bayesian RTM, these approaches can face challenges when scoring the innovative items.…”