Using a Theorem by Andersen and the Dichotomous Rasch Model to Assess the Presence of Random Guessing in Multiple Choice Items

Abstract

Andersen (1995, 2002) proves a theorem relating variances of parameter estimates from samples and subsamples and shows its use as an adjunct to standard statistical analyses. The authors show an application where the theorem is central to the hypothesis tested, namely, whether random guessing to multiple choice items affects their estimates in the Rasch model. Taking random guessing to be a function of the difficulty of an item relative to the proficiency of a person, the authors describe a method for creating a subsample of responses, which is least likely to be affected by guessing. Then using Andersen’s theorem, the authors assess the difference in difficulty estimates between responses from the whole sample and the subsample for each item. To demonstrate the effectiveness of the procedure, data are simulated according to a class of models in which random guessing is a function of the proficiency of a person relative to the difficulty of an item. The procedure is also applied to an empirical data set from Raven’s Advanced Progressive Matrices, with the results indicating that guessing is present in a substantial number of items. It is noted that one especially important application in which estimating the correct relative difficulty of items is required is where the items will form part of an item bank and where on subsequent occasions the items will be administered interactively. In this case, items too difficult for a person are not administered and therefore unlikely to attract random guessing.

An Application of the Rasch Measurement Theory to an Assessment of Geometric Thinking Levels

“…Person proficiency is then estimated in relation to item difficulty. For full details see Andrich (1988) and Dunne et al (2012). The map is essentially a crude histogram with items arranged on the right, from relatively easy at the bottom, to relatively difficult at the top.…”

“…The approach adopted here is that "guessing is a function of the difficulty of the item relative to the proficiency of the person" (Andrich, Marais & Humphry, 2011, p. 1). Andrich et al (2011) propose a strategy to "remove those responses most likely to be affected by guessing" (p. 3). These responses are removed and treated as a set of missing responses.…”

Using four-parameter item response theory to model human figure drawings

“…One important point is that these are strong assumptions that not always hold. See, for instance, Chiu and Camilli (2013), Andrich, Marais, and Humphry (2012). It will be interesting to study the implications of the use of 3PL model in scoring student responses on the Brazilian National Exam of High Education (ENEM).…”

Controlling Bias in Both Constructed Response and Multiple‐Choice Items When Analyzed With the Dichotomous Rasch Model

“…The same research has shown how the bias due to guessing in MC items when analyzed by the RM can be controlled (Andrich et al., , ). The importance of controlling the bias due to guessing in large‐scale assessments has both a direct and an indirect effect.…”