Assessment by Comparative Judgement: An Application to Secondary Statistics and English in New Zealand

Abstract

There is growing interest in using comparative judgement to assess student work as an alternative to traditional marking. Comparative judgement requires no rubrics and is instead grounded in experts making pairwise judgements about the relative 'quality' of students' work according to a high level criterion. The resulting decision data are fitted to a statistical model to produce a score for each student. Cited benefits of comparative judgement over traditional methods include increased reliability, validity and efficiency of assessment processes. We investigated whether such claims apply to summative statistics and English assessments in New Zealand. Experts comparatively judged students' responses to two national assessment tasks, and the reliability and validity of the outcomes were explored using standard techniques. We present evidence that the comparative judgement process efficiently produced reliable and valid assessment outcomes. We consider the limitations of the study, and make suggestions for further research and potential applications.

Assessing covariation as a form of conceptual understanding through comparative judgement

“…These time data are an overestimate because the timer of the online platform does not stop when judges move away from the computer leaving the screen open. The time required for the comparisons in our experimentation does not differ much from the estimate reported in Marshall et al (2020) and, in our opinion, it is reasonable when taking into consideration the length of texts, their nature, and difficulties linked to deciphering handwriting. However, based on our experience, the time is less than that usually required to grade a single text by a traditional marking method.…”

“…It is proven to be even more reliable than traditional marking in open-ended assessment not only in mathematics (Jones & Alcock, 2014;Steedle & Ferrara, 2016). This assessment technique has been used with several mathematical topics: problem-solving (Jones & Inglis, 2015), conceptual understanding (Bisson et al, 2016;Jones & Alcock, 2014;Jones & Karadeniz, 2016), mathematical proof (Davies et al, 2020), and also statistical knowledge (Bisson et al, 2016;Marshall et al, 2020). As far as we know, CJ has never been used to assess tests requiring covariational reasoning.…”

“…Although validation can never be fully demonstrated or completed (Marshall et al, 2020), suitable statistical methods allow us to verify the reliability and validity of CJ outcomes. All the details useful for the purposes of our research are provided in Sect.…”

“…The outcome showed high internal consistency since SSR ≥ 0.7 (SSR = 0.81). Moreover, looking for "misfitting" judges, we computed an infit statistic for every judge and compared this data to a threshold value, i.e., two standard deviations above the mean of the infit (Marshall et al, 2020;Pollitt, 2012). Only one judge resulted slightly above the threshold value (0.03 above); so we did not consider it necessary to adapt the CJ scores by removing the misfitting judge.…”

What makes a good educational research summary? A comparative judgement study of mathematics teachers' and mathematics education researchers' views

“…Recently, Sangwin and Kinnear (2021) also used CJ to judge several different aspects of mathematical proof, evaluating whether they were reliable constructs by post-hoc analysis. On the whole, CJ is considered effective in evaluation of complex, creative and composite work involving skills such as making connections between learned ideas, applying understanding to novel contexts, constructing arguments and demonstrating chains of reasoning (Marshall et al, 2020) which we suggest includes knowledge brokering work such as the research summaries evaluated in this study.…”