Controlling Guessing Bias in the Dichotomous Rasch Model Applied to a Large-Scale, Vertically Scaled Testing Program

Abstract

Recent research has shown how the statistical bias in Rasch model difficulty estimates induced by guessing in multiple-choice items can be eliminated. Using vertical scaling of a high-profile national reading test, it is shown that the dominant effect of removing such bias is a nonlinear change in the unit of scale across the continuum. The consequence is that the proficiencies of the more proficient students are increased relative to those of the less proficient. Not controlling the guessing bias underestimates the progress of students across 7 years of schooling with important educational implications.

Controlling Bias in Both Constructed Response and Multiple‐Choice Items When Analyzed With the Dichotomous Rasch Model

“…This is evident from a comparison of the effects on the assessment in reading from similar NAPLAN assessments reported in Andrich et al. (). In the reading assessments, the majority of items were MC.…”

“…Andrich et al. () interpret this difference in unit as a difference in precision—the tailored analysis, which does not include guessed responses, has greater precision than the origin‐equated analysis. However, the effect on the unit of scale is not linear across the continuum.…”

“…Clearly, the 3PL model specializes algebraically to the RM when and for all items. Figure (reproduced from Andrich et al., ) shows the item characteristic curves (ICCs) for the RM and the 3PL of an item with an asymptote for a totally random guess, = .25, which assumes four alternatives. We observe two points from the 3PL.…”

“…Andrich et al. () also investigated the effect of guessing on a vertical scale of reading proficiency that consisted mainly of MC items, with approximately 2% of CR items. The Reading assessments were part of NAPLAN which assesses all Australian students in Years 3, 5, 7, and 9.…”

“…Recent research has shown that the bias in the difficulty estimates of MC items from a RM analysis is systematic, nonlinear, and a function of the relative difficulty of an item—the more difficult the item, the greater the bias that reduces its relative difficulty (Andrich, Marais, & Humphry, , ). The reason that items of greater difficulty have greater bias is that the degree of guessing in the responses to an item is a function of its difficulty—the more difficult an item, the greater the degree of guessing.…”