The assessment of quantitative problem-solving skills with “none of the above”-items (NOTA items)

Abstract

In this contribution we concentrate on the features of a particular item format: items having as the last option "none of the above" (NOTA items). There is considerable dispute on the advisability of the usage of NOTA items in testing. Some authors come to the conclusion that NOTA items should be avoided, some come to neutral conclusions while others argue that NOTA items are optimal test items. In this article, we provide evidence to this discussion by conducting protocol analysis on written statements of examinees while answering NOTA items. In our investigation, a test containing 30 multiple-choice items was administered from 169 university students. The results show that NOTA options appear to be more attractive than options with specified solutions in those cases where a problemsolver fails. Also, a relationship is found between the quality of (incorrect) problemsolving and the choice of NOTA items: the more qualitative the incorrect problemsolving process is, the more likely the student is to choose for NOTA items. Overall, our research supports the statement that 'the more confidence an examinee has in his worked solution, which is inconsistent with one of the specified solutions, the more eager he seems to choose "none ofthe above".

Sources of measurement error in pediatric intelligence testing

“…Kubinger and Wolfsbauer (2010: 303) contend that from the point of view of personality psychology, examinees in a multiple-choice test are likely to differ in the way they deal with multiple choice items, which might impact their results. Dochy et al (2001) on their side presume that, if an examinee reaches a particular answer for an item in a multiple-choice test, which, however, is not offered as an option, then, the higher the examinee’s general assertiveness is, the more he/she uses “none of the other options is right.” Ávila and Torrubia (2004) established that extraverted, impulsive, and low-anxiety examinees give more incorrect responses in multiple-choice tasks but make fewer omission errors and Alker et al (1969) indicate that examinees with a “nonconformist” personality score higher on achievement tests than others because of fewer skipped items and not at all using the “I don’t know the solution” option. Lastly, Stoeber and Kersting (2007) showed that “perfectionist” examinees attain higher scores in achievement tests.…”

Creating Accurate Science Benchmark Assessments to Inform Instruction

“…As such, they can adversely affect the technical quality of the item because students who may know very little about the content of interest may be able to answer the question correctly. In addition, questions, especially complex questions, with "none of the above" as an answer, may disproportionately attract student responses because of minor misconceptions or procedural errors on the part of the student (Docy, Moerkerke, DeCorte, & Segers, 2001). • Choose distractors in order to pinpoint common misconceptions and provide educators information, which will be useful in follow-on instruction.…”

New Guidelines for Developing Multiple-Choice Items

“…In order to maintain the independence mentioned above, caution should be exercised in using the options “All of the above” and “None of the above.” Nevertheless, if it is decided to use them, it is important to bear in mind the following: The former appears to introduce an additional difficulty (Dudycha & Carpenter, 1973; Mueller, 1975), especially for subjects with low levels of knowledge (Martínez, Moreno, Martín, Trigo, & López, 2004), probably because it requires them to know that at least two of the above are correct. For its part, the option “None of the above” has a general difficulty effect (Dochy, Moerkerke, De Corte, & Segers, 2001;Haladyna et al, 2002), at least when it is constructed as the correct option (Martínez et al, 2004), probably because it involves negative language and logic, referring to what things are not, an indirect and normally more complicated form than referring to them in positive terms.…”