Theoretical evaluation of partial credit scoring of the multiple-choice test item

Abstract

In multiple-choice tests, guessing is a source of test error which can be suppressed if its expected score is made negative by either penalizing wrong answers or rewarding expressions of partial knowledge. Starting from the most general formulation of the necessary and sufficient scoring conditions for guessing to lead to an expected loss beyond the test-taker’s knowledge, we formulate a class of optimal scoring functions, including the proposal by Zapechelnyuk (Econ. Lett. 132, 24–27 (2015)) as a special case. We then consider an arbitrary multiple-choice test taken by a rational test-taker whose knowledge of a test item is defined by the fraction of the answer options which can be ruled out. For this model, we study the statistical properties of the obtained score for both standard marking (where guessing is not penalized), and marking where guessing is suppressed either by expensive score penalties for incorrect answers or by different marking schemes that reward partial knowledge.

Multiple-choice test items with "none of the above" as an alternative

“…As far as the undergirding assumptions of the model presented in Ref. [7] hold, the results presented here are certain. The good agreement with the majority of the experimental results thus serves to validate said assumptions and, by extension, the model.…”

“…Horst [20][21][22] seems to have been the first to present the idea in print that test-takers solve multiple-choice items by eliminating all options they know to be incorrect and then guess among the remaining ones. This general idea was taken up and formalized under some simplifying assumptions by Persson [7], and this is essentially the model on which we rely in this work. We shall extend the model slightly in Section 6 to include also the effect of misinformation.…”

“…The model of Ref. [7] is one where the correctness of each answer option (save for the potential "none-of-the-above" option) is assumed ascertainable independently of the others, and where the test-taker's ability is measured by the fraction f of the answer options for which she is a priori expected to be able to determine the correctness (again, not counting any "none-of-the-above" option). For brevity, we shall refer to the ascertainment of the correctness of a given option as the "recognition" of said option.…”

“…In fact, we may formalize Gross argument through a simple extension of the model of Ref. [7] as follows.…”

“…Building on a simple model [7] for the probability of solving a multiplechoice test item as a function the test-taker's "ability", we here investigate theoretically what others have studied empirically before us. We hope to shed light on the controversy and bring the matter to rest.…”