Rater strategies for reaching agreement on pupil text quality

Abstract

Novice members of a Norwegian national rater panel tasked with assessing Year 8 pupils' written texts were studied during three successive preparation sessions (2011)(2012). The purpose was to investigate how the raters successfully make use of different decision-making strategies in an assessment situation where preset criteria and standards give a rather strict framework. The data sources were the raters' pair assessment dialogues. The analysis shows that the raters use a 'shared standards strategy', but when reaching agreement on text quality they also seem to make very good use of assessment strategies related to their work as writing teachers. Moreover, asymmetries in knowledge and participation among raters contribute to creating an image of writing assessment as a challenging hermeneutic practice. It is suggested that future rater preparation would gain from being attentive to the internalised assessment practices teachers bring to the fore when working as raters.

Teachers as raters: Investigation of a long term writing assessment program

“…For example, the rater pair discussions might not have challenged the teachers' own understanding of text quality but, instead, might have served as social arenas for demonstrating perceptions rather than participating in negotiations (cf. Jølle, 2015). The NWC may have disproportionately stressed the teachers as experts.…”

“…Such effects can manifest through reliability estimates and other associated quantitative investigations (Borgström & Ledin, 2014) or by using qualitative data that demonstrate the ways in which raters use the assessment materials (e.g., rubrics and benchmark texts) as well as their perceptions of text quality (cf. Jølle, 2015). As previous research on communities of practice has shown (e.g., Nielsen, 2008), it is also important to include participants' perception of the purpose of being part of a community.…”

Validitätsaspekte bei der Bewertung schriftlicher Kompetenz

“…Erfahrene Lehrkräfte eines Faches scheinen zudem eine Vorstellung davon zu haben, was von Schülerinnen und Schülern unterschiedlicher Jahresstufen zu erwarten ist, und ihre Bewertungen nach diesen Maßstäben zu begründen (vgl. Jølle 2015). Dies könnte aber problematisch sein, da Lehrkräfte anscheinend nicht immer mit dem übereinstimmen, was von den Lernenden auf gleichem Niveau zu erwarten ist (vgl.…”

Är vi överens om det? Samspel och samstämmighet i svensklärares bedömningssamtal om gymnasieelevers skrivande

“…Berge (2002) har synliggjort och problematiserat brister i de tolkningsgemenskaper som utmärker norsklärares skrivbedömningspraktiker, något som följts av flertalet norska studier av beslutsprocesser i lärares skrivbedömningar. Exempelvis undersö-ker Jølle (2014Jølle ( , 2015 vilka strategier lärare använder vid parbedömning av elevers skrivande och hur lärares bedömarkompetens kan utvecklas i professionella tränings-program. Matre och Solheim (2015 undersöker norsklärares användning av bedömningsredskap samt hur lärare argumenterar och interagerar i samtal om skrivbedömning.…”