When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Abstract

Insight into aspects that guide teachers' decisions when assessing student text quality is crucial to an understanding of the validity of text scores. Such research has been lacking in the context of comparative methods which are, however, increasingly being used for text assessment purposes. This study reports on the aspects of argumentative texts that guide Flemish teachers' decisions when using the comparative judgement method. In using this method, teachers indicate which text of a pair is of higher quality. 27 teachers explained 23 comparative judgements in a decision statement, when comparing randomly selected texts written by 135 students in their fifth year of general secondary education. This resulted in 596 statements referring to 2054 segments of aspects of text quality. Firstly, an inductive analysis revealed that teachers consider a wide range of aspects with regard to text quality when making comparison decisions. Secondly, the deductive aggregation of these aspects showed that most decisions are informed by the organisation and argumentation of the texts. Lastly, almost all statements reported complex aspects of text quality, whereas half of the decision statements also showed a reflection on the rule-applying aspects of text quality. We conclude that comparative judgement encourages teachers to make decisions on complex and multiple aspects of text quality. Further research should elaborate on whether the aspects that informed teachers' decisions are related to the text they choose, and whether teachers differ in the aspects to which they refer.

A meta-analysis on the reliability of comparative judgement

“…However, it can be argued that including more assessors might increase the chance of having deviating assessors. Moreover, in her dissertation Lesterhuis (2018) has shown that assessors can differ in their opinion on aspects upon which representations are compared. This might influence the results, but can also increase the validity, with the assessment results providing more comprehensive image of the competence.…”

“…It can thus be stated that the SSR reflects the consistency between assessors in general. This could, however, also mean that judgements based on assessors who highly agree on what constitutes a good performance and on what are quality criteria for such performances, are associated with a higher SSR value and possibly also a lower validity (Lesterhuis, 2018; see also Lesterhuis et al, 2018;van Daal et al, 2016).…”

“…A second prominent characteristic of CJ is the number of assessors. One of the strengths of CJ is that it allows for, and even promotes, the use of multiple assessors, which increases the validity and generalizability of the assessment results (Lesterhuis et al, 2018;van Daal et al, 2017). It could also be argued, however, that engaging more assessors in the assessment leads to more differing perspectives on the particular competence under assessment, which might decrease the reliability.…”

Syntactic predictors for text quality in Dutch upper-secondary school students’ L1 argumentative writing

“…The latter has been criticized for being too systematic and constraining, thereby endangering the validity of the judgement (Sadler, 2009). Lesterhuis et al (2018) show that when argumentative texts are rated by means of comparative judgement, raters consider a wide range of relevant text quality aspects. They conclude from this that comparative judgement is a valid way to assess the quality of (argumentative) texts.…”

The Complexity of Comparative Judgments in Assessing Argumentative Writing: An Eye Tracking Study

“…Based on all comparative judgments, a ranking scale is generated, which orders the pieces of student work from lowest to highest quality (Pollitt, 2012a;Pollit, 2012b). This scale is assumed to reflect the consensus among the assessors regarding the quality of these texts (Jones and Alcock, 2014;Lesterhuis, 2018;Pollitt, 2012a;van Daal et al, 2019). Although research has demonstrated the reliability and validity of this method (e.g., Pollitt, 2012a;Pollit, 2012b;Jones and Inglis, 2015;Lesterhuis, 2018), it is questionable whether assessors can easily discriminate, decide which one is better, between all pairs of texts.…”