Validity of Comparative Judgment Scores: How Assessors Evaluate Aspects of Text Quality When Comparing Argumentative Texts

Abstract

The advantage of comparative judgment is that it is particularly suited to assess multidimensional and complex constructs as text quality. This is because assessors are asked to compare texts holistically and to make a quality judgment for each text in a pairwise comparison based upon on the most salient and critical differences. Also, the resulted rank order is based on the judgment of all assessors, representing the shared consensus. In order to be able to select the right number of assessors, the question is to what extent the conceptualization of assessors prevails in the aspects they base their judgment on, or whether comparative judgment minimizes the differences between assessors. In other words, can we detect types of assessors who tend to consider certain aspects of text quality more often than others? A total of 64 assessors compared argumentative texts, after which they provided decision statements on what aspects of text quality had informed their judgment. These decision statements were coded on six overarching themes of text quality: argumentation, organization, language use, language conventions, source use, references, and layout. Using a multilevel-latent class analysis, four different types of assessors could be distinguished: narrowly focused, broadly focused, source-focused, and language-focused. However, the analysis also showed that all assessor types mainly focused on argumentation and organization, and that assessor types only partly explained whether the aspect of text quality was mentioned in a decision statement. We conclude that comparative judgment is a strong method for comparing complex constructs like text quality. First, because the rank order combines different views on text quality, but foremost because the method of comparative judgment minimizes differences between assessors.

Comparative approaches to the assessment of writing: Reliability and validity of benchmark rating and comparative judgement

“…However, a recent meta-analysis on comparative judgement demonstrates that the format of the task (e.g., texts, images, video, or portfolios) hardly affects the level of reliability (Verhavert et al, 2019). It is also recommended to replicate the study with another sample of more experienced raters, even though previous studies showed no differences between student raters and more experienced raters for comparative judgement (Lesterhuis, 2018).…”

“…Furthermore, the total number of comparisons can be distributed among multiple raters, which decreases the workload per individual rater. Researchers have also shown that including multiple raters in a CJ session will increase the validity of the results, as the shared consensus of multiple raters appears to be a good reflection of all relevant aspects of the writing task (Jones & Alcock, 2014;Lesterhuis, 2018;Van Daal et al, 2019).…”

“…This might decrease the reliability of the scores in CJ in comparison to benchmark ratings, even though assessors in CJ can make a relatively easier decision (i.e., only pointing out the best text in a pair, instead of also assigning scores to the texts). In addition, even though results regarding the effects of rater training for the reliability of performance assessments are generally mixed (Lumley & McNamara, 1995;Rezaei & Lovorn, 2010), and assessors appear to focus predominantly on aspects that are related to the quality of the argumentation and organisation when comparing texts (Lesterhuis, 2018), it is yet unknown whether a training in CJ is required for assessors to keep their judgements consistent, and hence reliable, over time.…”

Comparative Judgement

“…To date, Comparative Judgement has proliferated across many writing assessment contexts, including large-scale primary and secondary-level assessment (e.g., McGrane et al, 2018;Pollitt, 2012;Wheadon, Barmby, Christodoulou, & Henderson, 2020), primary-level classroom-based assessment (e.g., Heldsinger & Humphry, 2010, 2013Humphry & Heldsinger, 2019b), higher education contexts (e.g., Lesterhuis, Bouwer, van Daal, Donche, & De Maeyer, 2022;van Daal, Lesterhuis, Donche, De Maeyer, & Coertjens, 2019), English as an additional language writing contexts (e.g., Şahin, 2021;Sims, Cox, Eckstein, Hartshorn, Wilcox, & Hart, 2020), as well as for the longitudinal equating of a standardised writing assessment (Humphry & McGrane, 2015).…”

“…In comparison to the issue of reliability, the validity of Comparative Judgement has much less attention, although there has recently been a proliferation of research focused on this issue (e.g., Chambers & Cunningham, 2022;Humphry & Heldsinger 2019a;Lesterhuis et al, 2022;Mentzer, Lee, & Bartholomew, 2021;van Daal, Lesterhuis, Coertjens, van de Kamp, Donche, & De Maeyer, 2017;van Daal et al, 2019). Typically, the method has been found to have moderate to high levels of concurrent validity with other more established methods like rubric-based ratings (e.g., Attali et al, 2014;Heldsinger & Humphry, 2010, 2013McGrane et al, 2018).…”