On Minimizing Guessing Effects on Multiple‐Choice Items: Superiority of a two solutions and three distractors item format to a one solution and five distractors item format

Abstract

Multiple‐choice response formats are troublesome, as an item is often scored as solved simply because the examinee may be lucky at guessing the correct option. Instead of pertinent Item Response Theory models, which take guessing effects into account, this paper considers a psycho‐technological approach to re‐conceptualizing multiple‐choice response formats. The free‐response format is compared with two different multiple‐choice formats: a traditional format with a single correct response option and five distractors (‘1 of 6’), and another with five response options, three of them being distractors and two of them being correct (‘2 of 5’). For the latter format, an item is scored as mastered only if both correct response options and none of the distractors are marked. After the exclusion of a few items, the Rasch model analyses revealed appropriate fit for 188 items altogether. The resulting item‐difficulty parameters were used for comparison. The multiple‐choice format ‘1 of 6’ differs significantly from the multiple‐choice format ‘2 of 5’, while the latter does not differ significantly from the free‐response format. The lower difficulty of items ‘1 of 6’ suggests guessing effects.

Interdisciplinary Research in History Education

“…Die Wahl des Antwortformats in Kombination mit der Kodierung kann u.U. einen (erheblichen) Einfl uss auf die Aufgabenschwierigkeit haben (Kubinger & Gottschall, 2007;Kubinger, Holocher-Ertl, Reif, Hohensinn & Frebort, 2010), die inhaltlich begründete Schwierigkeitsunterschiede relativieren.…”

I know that I know: A certainty based marking tests designed for evaluating knowledge of healthcare students

“…Nevýhodou této metody je vyšší obtížnost vyplňová-ní testu a větší časová náročnost. Zároveň klade vyšší nároky na tvorbu a kvalitu distraktorů (6).…”

“…O nevýhodách použití prostého testu s jednou správnou možností (ze šesti) v každé úloze píše i Kubinger (6). V jeho studii proběhlo porovnání oproti testu se dvěma správnými možnostmi a třemi distraktory.…”

Lessons learned from performance of students of Pharmacology in self coded surprise test with negative marking.

“…Therefore, guessing should neither be rewarded nor encouraged. Guessing also has a negative impact on the reliability of MCQs [5][6][7]. Varied scoring methods are available to accommodate for guessing [8,9].…”