A Two-Stage Method for Obtaining Reliable Teacher Assessments of Writing

Abstract

In many countries, systems have employed externally imposed large-scale standardized assessments in seeking to obtain reliable and comparable assessments. To attain the same objectives while placing value on professional judgement, appropriate methods of assessment for classroom teachers need to be developed and tested. Accordingly, this paper examines the level of reliability of assessments made by classroom teachers of narrative writing using a two-stage classroom assessment method. The students involved in the study are primary students. The results show high levels of inter-rater reliability among teachers. The findings reproduce and expand on previous evidence indicating the two-stage method is a viable method of classroom teacher assessment of written performances. Implications of the results for assessment practices in education are discussed in light of background literature, with a focus on the widely expressed desire to value professional expertise.

Using pairwise comparison and ordered exemplars as a basis for a novel method of standard setting in narrative writing

“…From the original pool of 625 performances, 164 performances from Queensland and 115 performances from Western Australia (together totaling 279 performances) were selected for use in Stage 1. Performances were selected based on several criteria including approximately equal distribution of school awarded A to E grades, 2 A detailed description of the performances, judges, and data analysis involving the WA performances from five of the schools is provided in Humphry and Heldsinger (2020). The present study included performances from two additional schools.…”

“…The standard error (SE) of the scale location of a performance is dependent on the number of comparisons as well as the relative locations of the performances it is compared against (Humphry and Heldsinger, 2020). For this reason, the maximum number of comparisons per performance is sought given the available time from judges and the anticipated number of comparisons that can be made in that time.…”

“…These comparative judgments were combined with 3,532 comparative judgments of the original 160 Western Australian narrative performances (Humphry and Heldsinger, 2020), resulting in a total of 6,541 pairwise comparisons of 324 performances to be analyzed.…”

Group Moderation Assessment Model: An Example of an Open-Ended Mathematics Exam

“…Reliability as defined by Kerlinger (1992) is the accuracy or precision of a measuring instrument. Reliability testing determines if an instrument is free from error and provides consistent results (Association for Advanced Training, 1988;Humphry and Heldsinger, 2020). The more reliable a test, the less error there is in the test.…”

“…Measurement results close to real scores are realized in the extent of the low error scores in the measurements. In other words, measurement results are reliable to the extent that they have few random errors (Baykul, 2000;Humphry and Heldsinger, 2020).…”