How accurate are examiners’ holistic judgements of script quality?

Gill, Tim; Bramley, Tom

doi:10.1080/0969594x.2013.779229

Cited by 19 publications

(20 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, judges experience more self-confidence in their decisions for some CJs than for others. As self-confidence was found to positively relate to decision accuracy (Gill and Bramley, 2013), this underpins the significance of judges' experienced complexity for judgment quality. The latter study also incorporates the second perspective on complexity: complexity as an objective characteristic of the judgment task.…”

mentioning

confidence: 99%

The Complexity of Assessing Student Work Using Comparative Judgment: The Moderating Role of Decision Accuracy

et al. 2017

View full text Add to dashboard Cite

Nowadays, comparative judgment (CJ) is used to assess competences. Judges compare two pieces of student work and judge which of both is better regarding the competence assessed. Using these pairwise comparison data, students' work is scaled according to its quality. Since student work is highly information loaded and heterogeneous of nature, this raises the question whether judges can handle this type of complex judgments? However, research into the complexity of CJ and its relation with decision accuracy is currently lacking. Therefore, this study initiates a theoretical framework on the complexity of CJ and relates it to decision accuracy. Based on this framework, two hypotheses are formulated and their plausibility is examined. The first hypothesis states that the distance between two pieces of student work on the rank-order (rank-order distance) is negatively related to experienced complexity, irrespectively of decision accuracy. In contrast, hypothesis 2 expects decision accuracy to moderate the relation between rank-order distance and experienced complexity. A negative relation is expected for accurate decisions. Meanwhile, inaccurate decisions are assumed to result in higher experienced complexity than accurate decisions, irrespective of rank-order distance. In both hypotheses, judges are expected to vary in mean experienced complexity as well as in the strength of the expected relationship between rank-order distance and experienced complexity. Using an information-theoretic approach, both hypotheses are translated into a statistical model and their relative fit is assessed. All analyses are replicated on three samples. Sample 1 and 2 comprise CJ data on the assessment of writing, while sample 3 contains pairwise comparison data on the assessment of visual arts. In all samples, results unambiguously confirm the moderating role of decision accuracy (hypothesis 2). Inaccurate decisions are experienced as more complex than accurate decisions, irrespective of rank-order distance. Meanwhile, for accurate decisions, rank-order distance is negatively related to experienced complexity. In line with expectations, differences between judges are found in mean experienced complexity and in the strength of the relationship between rankorder distance and experienced complexity. Suggestions for further theory development are formulated based on the results of this study.

show abstract

mentioning

confidence: 99%

The Complexity of Assessing Student Work Using Comparative Judgment: The Moderating Role of Decision Accuracy

et al. 2017

View full text Add to dashboard Cite

show abstract

“…The quality of each text is then calculated by analyzing all the comparative judgements made by all assessors (more information on the method can be found in Lesterhuis, Verhavert, Coertjens, Donche & De Maeyer, 2016, and Pollitt, 2012a, 2012b. Given that comparing texts is an easier and more reliable task when it comes to assessing complex issues such as text quality (Gill & Bramley, 2013), comparative methods have been increasingly appraised (Bramley, 2007;Heldsinger & Humphry, 2010;Pollitt, 2012a;Steedle & Ferrara, 2016). However, little is known about which aspects of text quality assessors consider when comparing texts, which hampers a valid interpretation of the assessment outcomes.…”

Section: Introductionmentioning

confidence: 99%

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Lesterhuis¹,

Daal²,

Gasse³

et al. 2018

L1 Educational Studies in Language and Literature

View full text Add to dashboard Cite

Insight into aspects that guide teachers' decisions when assessing student text quality is crucial to an understanding of the validity of text scores. Such research has been lacking in the context of comparative methods which are, however, increasingly being used for text assessment purposes. This study reports on the aspects of argumentative texts that guide Flemish teachers' decisions when using the comparative judgement method. In using this method, teachers indicate which text of a pair is of higher quality. 27 teachers explained 23 comparative judgements in a decision statement, when comparing randomly selected texts written by 135 students in their fifth year of general secondary education. This resulted in 596 statements referring to 2054 segments of aspects of text quality. Firstly, an inductive analysis revealed that teachers consider a wide range of aspects with regard to text quality when making comparison decisions. Secondly, the deductive aggregation of these aspects showed that most decisions are informed by the organisation and argumentation of the texts. Lastly, almost all statements reported complex aspects of text quality, whereas half of the decision statements also showed a reflection on the rule-applying aspects of text quality. We conclude that comparative judgement encourages teachers to make decisions on complex and multiple aspects of text quality. Further research should elaborate on whether the aspects that informed teachers' decisions are related to the text they choose, and whether teachers differ in the aspects to which they refer.

show abstract

“…Absolute and relative judgements are terms that have been used to describe analytical and comparative judgement assessment methods respectively. Absolute judgements, analytical traditional marking/scoring, refers to the allocation of marks/score points or a grade (Gill & Bramley, 2013;Pollitt, 2012a). It has been contended by researchers that relative, comparative judgements are highly reliable, more so than absolute, analytical judgements (see Jones & Alcock, 2012Laming, 2004;Pollitt, 2012b).…”

Section: Relative and Absolute Judgementsmentioning

confidence: 99%

“…The recent use of these digital comparative judgment systems have been for standard setting (Bramley, 2007;Bramley et al, 1998;Gill & Bramley, 2013), moderation and marking of scripts and portfolios in areas such as writing (Heldsinger & Humphry, 2010Pollitt, 2012b), mathematical problem solving (Jones et al, 2015), science (McMahon & Jones, 2014), design and visual arts (Newhouse & Tarricone, 2014), design and technology e-portfolio assessment (Kimbell et al, 2009), scientific and technology enquiry e-portfolio assessment (Davies et al 2012), oral language assessment (Newhouse, 2011;Pollitt & Murray, 1996), and peer assessment (Jones & Alcock, 2012Seery et al 2012). These studies have provided evidence of the efficacy in using digital assessment systems for relative comparative judgement.…”

Section: Use Of Technology and Comparative Judgementmentioning

confidence: 99%

“…Pollitt has also consistently supported the argument that comparative judgements, specifically adaptive comparative judgements, can generate "extremely reliable scores, far higher than traditional marking" with little training (Pollitt, 2012a, p. 168). For example, a study investigating absolute and relative judgements and standard-setting by Gill and Bramley (2013) considered the accuracy of holistic comparative judgements in relation to the original analytical marks of examination scripts that were 'close together in overall mark' in a standard-setting exercise. The assessors made absolute (mark/grade of each script), relative (script quality compared to another script) and confidence judgements (how confident they were of their judgements).…”

Section: Relative and Absolute Judgementsmentioning

confidence: 99%

See 1 more Smart Citation

Using comparative judgement and online technologies in the assessment and measurement of creative performance and capability

Tarricone

Newhouse

2016

Int J Educ Technol High Educ

View full text Add to dashboard Cite

In this paper we argue that comparative judgement delivered by online technologies is a viable, valid and highly reliable alternative to traditional analytical marking. In the past, comparative judgement has been underused in educational assessment and measurement, particularly in large-scale testing, mainly due to the lack of supporting technologies to facilitate the large number of judgements and judges. We describe the foundations of comparative judgement and dispel many of the old issues regarding its use in regards to time, cost and training for large-scale assessment. Studies in the use of comparative judgement and online technologies for the assessment and measurement of practical performance conducted by Edith Cowan University provide a context for further promoting its use in educational testing.

show abstract

How accurate are examiners’ holistic judgements of script quality?

Cited by 19 publications

References 6 publications

The Complexity of Assessing Student Work Using Comparative Judgment: The Moderating Role of Decision Accuracy

The Complexity of Assessing Student Work Using Comparative Judgment: The Moderating Role of Decision Accuracy

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Using comparative judgement and online technologies in the assessment and measurement of creative performance and capability

Contact Info

Product

Resources

About