Validity of comparative judgement to assess academic writing: examining implications of its holistic character and building on a shared consensus

Abstract

Recently, comparative judgement has been introduced as an alternative method for scoring essays. Although this method is promising in terms of obtaining reliable scores, empirical evidence concerning its validity is lacking. The current study examines implications resulting from two critical assumptions underpinning the use of comparative judgement, namely its holistic characteristic and how the final rank-order reflects the shared consensus on what makes for a good essay. Judges' justifications that underpin their decisions are qualitatively analysed to obtain insight into the dimensions of academic writing they take into account. The results show that most arguments are directly related to the competence description. However, judges also use their expertise in order to judge the quality of essays. Additionally, judges differ in terms of how they conceptualize writing quality, and regarding the extent to which they tap into their own expertise. Finally, this study explores divergence conceptualisation of misfitting judges.

The Complexity of Assessing Student Work Using Comparative Judgment: The Moderating Role of Decision Accuracy

“…Nowadays, CJ is used to assess competences such as writing (Pollitt, 2012a, van Daal et al, 2016 and mathematical understanding (Jones et al, 2015). Consequently, judges are asked to compare for example students' essays or portfolios (representations) and decide which is of higher quality.…”

“…These topics were "Getting children" (sample 1) and "Organ donation" (sample 2). The assignments were used in previous studies (Van Weijen, 2009;Tillema, 2012) and slightly adapted for use in the present study. Students were given 25 min to respond to each prompt and gave their written consent for inclusion of their essays for research purposes.…”

“…inTrODUcTiOn Recently, comparative judgment (CJ) was introduced into the field of educational measurement to assess competences, such as mathematical understanding (Jones et al, 2015), geography (Whitehouse and Pollitt, 2012), design and technology (Seery et al, 2012), and writing (Pollitt, 2012a;van Daal et al, 2016). Judges are presented with two pieces of student work (e.g., essays or portfolio's) and are asked to judge which piece shows more of the competence assessed.…”

A Two‐Stage Method for Classroom Assessments of Essay Writing

“…Much of the research focuses on applications of pairwise comparisons to produce scales in a range of areas. Perhaps the most focus to date has been on judging and scaling students' writing performances (Heldsinger & Humphry, 2010;Steedle & Ferrara, 2016;van Daal, Lesterhuis, Coertjens, Donche, & De Maeyer, 2016) and students' problem solving in mathematics (Bisson, Gilmore, Inglis, & Jones, 2016;Jones & Inglis, 2015;Jones, Swan, & Pollitt, 2015). Other areas of focus have included peer assessment and feedback (Potter et al, 2017;Seery, Canty, & Phelan, 2012), the assessment of creative performances (Tarricone & Newhouse, 2016), and the assessment of oral narrative performances (Humphry, Heldsinger, & Dawkins, 2017).…”

Untitled

“…van Daal, Lesterhuis, and Coertjens (2016), who looked at the assessment of academic writing, utilized professionals with varied expertise in the writing department for their assessors. Each of these individuals had background and experience in assessment writing products.…”