A meta-analysis on the reliability of comparative judgement

Verhavert, San; Bouwer, Renske; Donche, Vincent; Mæyer, Sven De

doi:10.1080/0969594x.2019.1602027

Cited by 41 publications

(59 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The internal consistency of the comparative judgement scores was strong, SSR = 0.85 and the split-halves inter-rater reliability was moderate, r = 0.72. The reliability of comparative judgement scores is related to the number of judgements (Verhavert et al 2019), and the lower values here compared to Study 1 (SSR = 0.90, r = 0.81) would be expected because there were fewer judgements per script. There were no misfitting judges for the case of the English assessment.…”

Section: Reliabilitymentioning

confidence: 55%

“…The internet has enabled larger scale of usage than is possible in traditional laboratory settings because student work can be digitised and presented to examiners remotely and efficiently (Pollitt 2012). A key motivation for considering comparative judgement rather than traditional assessment methods are claims that comparative judgement is more reliable than marking for the case of open-ended assessments (Jones and Alcock 2014;Steedle and Ferrara 2016;Verhavert et al 2019).…”

Section: Comparative Judgementmentioning

confidence: 99%

“…Once the judging is complete the examiners' binary decisions are fitted to a statistical model to produce a unique score for each script (Bramley 2007). It is not necessary to compare every script with every other script; the literature suggests that to assess n scripts then a total 10n to 37n judgements are required (Jones and Alcock 2014;Verhavert et al 2019), depending on the nature of the tests and the expertise of the assessors, to produce a reliable scale. The reliability and validity of outcomes can be estimated using the techniques described in the methods section.…”

Section: Comparative Judgementmentioning

confidence: 99%

See 2 more Smart Citations

Assessment by Comparative Judgement: An Application to Secondary Statistics and English in New Zealand

Marshall¹,

Shaw²,

Hunter

et al. 2020

NZ J Educ Stud

View full text Add to dashboard Cite

There is growing interest in using comparative judgement to assess student work as an alternative to traditional marking. Comparative judgement requires no rubrics and is instead grounded in experts making pairwise judgements about the relative 'quality' of students' work according to a high level criterion. The resulting decision data are fitted to a statistical model to produce a score for each student. Cited benefits of comparative judgement over traditional methods include increased reliability, validity and efficiency of assessment processes. We investigated whether such claims apply to summative statistics and English assessments in New Zealand. Experts comparatively judged students' responses to two national assessment tasks, and the reliability and validity of the outcomes were explored using standard techniques. We present evidence that the comparative judgement process efficiently produced reliable and valid assessment outcomes. We consider the limitations of the study, and make suggestions for further research and potential applications.

show abstract

Section: Reliabilitymentioning

confidence: 55%

Section: Comparative Judgementmentioning

confidence: 99%

Section: Comparative Judgementmentioning

confidence: 99%

See 1 more Smart Citation

Assessment by Comparative Judgement: An Application to Secondary Statistics and English in New Zealand

Marshall¹,

Shaw²,

Hunter

et al. 2020

NZ J Educ Stud

View full text Add to dashboard Cite

show abstract

“…In recent times, comparative judgment (CJ) has been introduced to assess competences, such as for example writing (e.g., Pollitt, 2012a;Pollit, 2012b;Verhavert, Bouwer, Donche, & De Maeyer, 2019). Assessors compare two texts and make relative, holistic judgments.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

The Complexity of Comparative Judgments in Assessing Argumentative Writing: An Eye Tracking Study

et al. 2021

Self Cite

View full text Add to dashboard Cite

Comparative judgment (CJ) has been recently introduced in the educational field as a means of assessing competences. In this judgement process, assessors are presented with two pieces of student work and are asked to choose which one is better in relation to the competencies being assessed. However, since student work is heterogeneous and highly information loaded, it raises the question as to whether this type of assessment is too complex for assessors to use. Previous research on the topic has operationalized experienced complexity by employing self-report measures, which have been criticized for common problems associated with their use. In our study, we used eye tracking to study 23 high school teachers when they made 10 comparative judgments, and their pupil diameter was used as an indicator of the experienced complexity. This study builds on previous research that integrated Campbell’s theory on task complexity (1988) into CJ. Based on this framework, three hypotheses regarding the role of decision accuracy were formulated and empirically tested. Hypothesis one assumes that the distance between two pieces of student work on the rank-order (rank-order distance) is negatively related to experienced complexity, irrespective of decision accuracy. Hypothesis two assumes that decision accuracy moderates the relationship between rank-order distance and experienced complexity. Hypothesis three builds on hypothesis two by adding a negative relationship between experience and experienced complexity. In all three hypotheses, the average experienced complexity is assumed to vary between assessors, as is the strength of the expected relationships. An information-theoretic approach was used to test the holding of all three hypotheses. All hypotheses were translated into statistical models, and their relative and absolute fit were assessed. Results provided strong evidence for hypothesis three: both the moderating role of decision accuracy on the relationship between rank-order distance and experienced complexity, and the relationship between experience and experienced complexity, were confirmed.

show abstract

“…They reported that mathematicians were reliable (consistent with one another) when making pairwise decisions about students' proof summaries. Good reliability has often been reported when CJ is applied to educational assessment across different subjects and contexts (Verhavert et al, 2019), and was the initial motivation for Davies et al to apply it to proof comprehension assessment.…”

mentioning

confidence: 99%

Assessing Proof Reading Comprehension Using Summaries

Davies

Jones

2021

Int. J. Res. Undergrad. Math. Ed.

View full text Add to dashboard Cite

In this paper, we explore the role of mathematical proof summaries as a tool for capturing students’ reading comprehension of a given proof. We present an interview study based on mathematicians’ pairwise evaluations of student-produced summaries of a proof demonstrating the uncountability of the open unit interval. We present a thematic analysis, exploring features of mathematicians’ pairwise decision-making and their priorities in evaluating summaries. We argue that the students’ proof summaries shared several properties with traditional modes of proof-writing and were frequently evaluated against similar conventions. We consider the consequences for research and practice with proof comprehension and conclude that proof summaries have the potential to form the basis of a new approach to assessment in this area.

show abstract

A meta-analysis on the reliability of comparative judgement

Cited by 41 publications

References 29 publications

Assessment by Comparative Judgement: An Application to Secondary Statistics and English in New Zealand

Assessment by Comparative Judgement: An Application to Secondary Statistics and English in New Zealand

The Complexity of Comparative Judgments in Assessing Argumentative Writing: An Eye Tracking Study

Assessing Proof Reading Comprehension Using Summaries

Contact Info

Product

Resources

About