Person Proficiency Estimates in the Dichotomous Rasch Model When Random Guessing Is Removed From Difficulty Estimates of Multiple Choice Items

Andrich, David; Marais, Ida

doi:10.1177/0146621614529646

Cited by 11 publications

(9 citation statements)

References 14 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One procedure to reduce the effects of guessing on parameter recovery in the Rasch model is to code responses as missing when the initial estimate of the item difficulty is considerably higher than the initial estimate of the person ability (Andrich & Marais, 2014; Andrich, Marais, & Humphry, 2012; Choppin, 1983), a procedure labeled tailored calibration. The rationale is that this removes the responses most likely to be affected by guessing.…”

mentioning

confidence: 99%

“…When the mean location of the easiest items was set equal across the two models, the negative bias in the other item locations increased with increasing difficulty. The same procedure was followed by Andrich and Marais (2014), who showed that the bias in item parameters produced corresponding bias in person estimates if likely guesses were not recoded as missing. High-ability examinees were penalized because the difficulty of the most difficult items was underestimated.…”

mentioning

confidence: 99%

See 1 more Smart Citation

The Interaction of Ability Differences and Guessing When Modeling Differential Item Functioning With the Rasch Model

DeMars

Jurich

2014

Educational and Psychological Measurement

View full text Add to dashboard Cite

In educational testing, differential item functioning (DIF) statistics must be accurately estimated to ensure the appropriate items are flagged for inspection or removal. This study showed how using the Rasch model to estimate DIF may introduce considerable bias in the results when there are large group differences in ability (impact) and the data follow a three-parameter logistic model. With large group ability differences, difficult non-DIF items appeared to favor the focal group and easy non-DIF items appeared to favor the reference group. Correspondingly, the effect sizes for DIF items were biased. These effects were mitigated when data were coded as missing for item-examinee encounters in which the person measure was considerably lower than the item location. Explanation of these results is provided by illustrating how the item response function becomes differentially distorted by guessing depending on the groups' ability distributions. In terms of practical implications, results suggest that measurement practitioners should not trust the DIF estimates from the Rasch model when there is a large difference in ability and examinees are potentially able to answer items correctly by guessing, unless data from examinees poorly matched to the item difficulty are coded as missing.

show abstract

mentioning

confidence: 99%

mentioning

confidence: 99%

The Interaction of Ability Differences and Guessing When Modeling Differential Item Functioning With the Rasch Model

DeMars

Jurich

2014

Educational and Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…One concern that has been raised regarding the development of multiple-choice tests relates to the plausibility of the distracters (Andrich and Marais, 2014). Poor or implausible distracter options can be easily rejected allowing the participant to choose the correct response by default.…”

Section: Experienced Novicementioning

confidence: 99%

The hazard prediction test: A comparison of free-response and multiple-choice formats

Ventsislavova

2018

Safety Science

View full text Add to dashboard Cite

Hazard perception skill is often related to lower crash risk, and the hazard perception test has been widely employed to measure this ability in drivers. An increasingly popular test-variant is the hazard prediction test: driving videos are occluded immediately prior to a hazard and participants are asked to predict how the situation will develop. Early versions of this test asked participants to provide a free-response answer which was subsequently coded. Later versions, however, have used a multiple-choice format where participants are provided with four options presented on screen. While the benefits of a multiple-choice format are obvious in terms of providing immediate feedback without relying on subjective coding, it is unclear whether this change in format affects the discriminative validity of the test. For the current study, a free-response test and a multiple-choice test were created using the same video clips. The free-response test (experiment 1) was found to successfully discriminate between novice and experienced drivers, with the latter predicting more hazards correctly. The answers provided by participants in Experiment 1 were then used to generate the options for a multiple-choice test (experiment 2). This second test was also found to discriminate between novice and experienced drivers, and a comparison between the two tests failed to reveal an advantage for one over the other. Despite this, correlations between prediction accuracy and both years of post-license driving, and annual mileage, were only significant for the multiple-choice test. The results suggest that the multiple-choice format is not only time-and cost-efficient, but is ostensibly as good as the free-response test in discriminating between driver groups.

show abstract

“…As indicated in the introduction, in the studies by Andrich et al. () and Andrich and Marais () the data sets contained responses to MC items only and did not contain any CR items. However, to better assess students’ knowledge and skills, assessments often contain multiple item formats, frequently a combination of MC items and CR items.…”

Section: The Dichotomous Rm and The Control Of Guessingmentioning

confidence: 99%

Controlling Bias in Both Constructed Response and Multiple‐Choice Items When Analyzed With the Dichotomous Rasch Model

Andrich

Marais

2018

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

Even though guessing biases difficulty estimates as a function of item difficulty in the dichotomous Rasch model, assessment programs with tests which include multiplechoice items often construct scales using this model. Research has shown that when all items are multiple-choice, this bias can largely be eliminated. However, many assessments have a combination of multiple-choice and constructed response items. Using vertically scaled numeracy assessments from a large-scale assessment program, this article shows that eliminating the bias on estimates of the multiple-choice items also impacts on the difficulty estimates of the constructed response items. This implies that the original estimates of the constructed response items were biased by the guessing on the multiple-choice items. This bias has implications for both defining difficulties in item banks for use in adaptive testing composed of both multiple-choice and constructed response items, and for the construction of proficiency scales.

show abstract

Person Proficiency Estimates in the Dichotomous Rasch Model When Random Guessing Is Removed From Difficulty Estimates of Multiple Choice Items

Cited by 11 publications

References 14 publications

The Interaction of Ability Differences and Guessing When Modeling Differential Item Functioning With the Rasch Model

The Interaction of Ability Differences and Guessing When Modeling Differential Item Functioning With the Rasch Model

The hazard prediction test: A comparison of free-response and multiple-choice formats

Controlling Bias in Both Constructed Response and Multiple‐Choice Items When Analyzed With the Dichotomous Rasch Model

Contact Info

Product

Resources

About