Distractor Functioning in Modified Items for Test Accessibility

Abstract

Effective distractors in multiple-choice items should attract lower ability students, those with misconceptions or limited knowledge and skills, because they are based on common misconceptions or errors in logic. A large, multi-state data set collected for a quasi-experimental study on test modifications was analyzed to measure the impact on distractor functioning. The key modification of interest was the removal of the weakest of three distractors, from 39 items in reading and 39 items in mathematics. Distractor functioning was neither systematically improved nor systematically weakened through the modification process. However, more than 70% of the distractors became more discriminating. A moderate correlation between distractor selection rate and distractor discrimination, in mathematics, may have indicated that the modified items were being missed by the appropriate students. Implications of these findings for test developers are discussed.

A Simulation‐Based Method for Finding the Optimal Number of Options for Multiple‐Choice Items on a Test

“…If maintaining test reliability is the ultimate goal, the discrimination criterion is recommended for removing nonfunctioning options. This practice was also recommended in previous studies (Lord, ; Rodriguez et al, ; Williams & Ebel, ).…”

“…If maintaining test reliability is the ultimate goal, the discrimination criterion is recommended for removing nonfunctioning options. This practice was also recommended in previous studies (Lord, 1980;Rodriguez et al, 2014;Williams & Ebel, 1957). .39 .22 .48 .12 N7m .43 .21 .48 .11 N6m .46 .21 .47 .11 N5m .49 .20 .48 .10 N4m .50 .21 .49 .09 N3m .58 .18 .45 .09…”

“…For newly generated multiple‐choice questions, test writers intentionally avoid three‐option questions by discarding any questions that have only three plausible options or by adding fillers such as “all of the above” and “none of the above.” These item‐writing practices are undesirable because they introduce construct‐irrelevant variance into the assessment (some examinees may be relatively more or less prone to select “all of the above” or “none of the above” independently of item content). Rodriguez et al () used a large multistate data set to measure the impact on distractor functioning. Distractor functioning was neither systematically improved nor weakened by reducing the number of options.…”

“…Studies have also explored the effects of changing the number of response options on item quality empirically (Baghaeri & Amrahi, 2011;Rodriguez, Kettler, & Elliott, 2014;Schneid, Armour, Park, Rudkowsky, & Bordage, 2014). Using a common-item equating design approach, Baghaeri and Amrahi (2011) found that, with one exception, no significant change was observed in item difficulties, item fit statistics, and reliabilities across three test forms with different numbers of options.…”

“…These item-writing practices are undesirable because they introduce construct-irrelevant variance into the assessment (some examinees may be relatively more or less prone to select "all of the above" or "none of the above" independently of item content). Rodriguez et al (2014) used a large multistate data set to measure the impact on distractor functioning. Distractor functioning was neither systematically improved nor weakened by reducing the number of options.…”