The Reliability of Marking of General Certificate of Secondary Education Scripts: mathematics and English

Abstract

This study investigated the consistency with which marks were awarded to scripts during the summer 1994 examinations in mathematics and English. It was demonstrated that the reliability of marking of mathematics scripts was extremely high. The consistency of marking in English was notably lower, although on the whole it appeared to be at least as good, if not better, than in previous studies. As a consequence of inconsistency in marking, the adjustment of Assistant Examiners’ marks is not uncommon for English. An investigation into this process demonstrated its validity. In comparison with mathematics, should we conclude that the examination boards are failing in their assessment of English? An alternative argument was proposed, claiming that any examining body must trade reliability of assessment against considerations of validity and cost‐effectiveness. Although problems of inconsistency arise in English, they are largely inevitable as long as we value the current assessment formats.

Peer assessment without assessment criteria

“…Indeed, the expert inter-rater reliability was higher than might be expected for an unstructured written task assessed against criteria (e.g. Murphy 1982;Newton 1996). This finding stands in contrast to empirical results reported in the peer assessment literature, which support a correlation between the use of clearly understood criteria and good validity and reliability.…”

“…Second, expert assessments can be unreliable (Falchikov and Goldfinch 2000;Magin 2001b;Murphy 1982;Newton 1996;Topping 2003), so the assumption that they constitute an objective reference may not be warranted. To address this, our first step in this study was to measure the expert inter-rater reliability of using CJ assess advanced mathematical understanding.…”

Asymmetry in student achievement on multiple-choice and constructed-response items in reversible mathematics processes

“…One option is to use CR formats, which are relatively easy to score reliably (Newton, 1996), or can be implemented online and scored automatically (Sangwin, 2013), as was the case here.…”

Marking as judgment

“…where there is a substantial element of uncertainty inherent in the subject and/or assessment method (Newton 1996;Wilmut, Wood, and Murphy 1996). Moreover, a wide range of biases has been detected in educational assessment.…”