Scoring rules and probability testing

Abstract

When using multiple-choice items in educational testing, the examinee is usually forced to choose exactly one ofthe answer possibilities as the supposedly correct answer. An alternative response mode is to let examinees assign probabilities to each of the answers of the item, corresponding to their belief in the correctness of each answer. In this study student attitudes toward this response mode were investigated, and scoring rules for assessing item scores were compared. It appeared that students had no clear preference for either us ing subjective probabilities or making forced choices, and that from a psychometrical point of view scoring rules based only on the probability assigned to the correct answer have to be preferred to scoring rules using all probabilities assigned to each of the answers.In any form of psychological or educational testing, a distinction can be made between response mode and scoring rule . The response mode is the way in which the subject is required to respond to the stimulus material. The scoring rule prescribes how numerical values are assigned to the responses. When multiple-choice examinations are used to measure a specific ability , the common response mode is the forced choice; the examinee is requested to mark precisely one of the answer possibilities as the (supposedly) correct one. The usual scoring rule assigns an item score of one to the examinee if the correct answer has been marked and an item score of zero if not. The test score is then taken to be the sum of the item scores .The combination of this forced-choice response mode and dichotomous scoring rule has the advantage of a quick and efficient computerized processing of large numbers of examinations. On the other hand, there are two disadvantages. The forced-choice response mode introduces error variance into the test scores because of guessing, and the dichotomous scoring rule does not discriminate between different levels of partial knowledge. Both of these factors are a potential threat to the reliability of the test.One of the possible ways of overcoming these disadvantages is to use personal probabilities as a response mode to multiple-choice items. In its purest form, this response mode requires assigning a personal probability to each of the answers of a test item, according to the degree to which the answer is believed to be the correct one . Per item, these probabilities should add to unity. De Finetti (1965) gave an extensive overview of the different varieties of the response mode based on personal probabilities.The personal probability response mode requires a specific scoring rule , a function assigning a real number to an item based on the distribution of the personal probabilities over the answers of the item. Van Naerssen