Rater bias patterns in an EFL writing assessment

Abstract

The present study employed multi-faceted Rasch measurement (MFRM) to explore the rater bias patterns of native English-speaker (NES) raters when they rate EFL essays. Forty NES raters rated 40 essays written by female Japanese university students on a single topic adapted from the TOEFL Test of Written English (TWE). The essays were assessed using a six-category rating scale (Content, Organization, Style and Quality of Expression, Language Use, Mechanics, and Fluency). MFRM revealed several recurring bias patterns among rater subgroups. In rater—category bias interactions, if Content and/or Organization were rated severely, then Language Use and/or Mechanics were rated leniently, and vice versa. In rater—writer bias interactions, there tended to be more severe or lenient bias towards higher ability writers than lower ability writers. Some raters also rated higher ability writers more severely and lower ability writers more leniently than expected. This study has implications for issues of rater training in L2 writing assessment.

Investigating the Effect of Training on Raters’ Bias toward Test Takers in Oral Proficiency Assessment : A FACETS Analysis

“…This indicates that the raters demonstrate more severity in rating when rating highly competent test takers; however, they were fairly lenient in their ratings toward extremely weak test takers. This finding is parallel, albeit in a writing assessment test, to one found by Schaefer (2008), who in an analysis of ratings by 40 native English speakers of 40 essays by Japanese students found some raters scored higher ability test takers more severely and lower ability ones more leniently than expected. The reason of this interaction tendency is not quite clear; however, it might be due to the fact that raters' expectations of test takers rise as test takers' abilities increase, thus making their judgments severer.…”

Measuring Essay Assessment: Intra-rater and Inter-rater Reliability

“…Assessing writing ability and the reliability of ratings have been a challenging concern for decades and there is always variation in the elements of writing preferred by raters and there are extraneous factors causing variation (Blok, 1985;Chase, 1968;Chase, 1983;Darus, 2006;East, 2009;Engelhard, 1994;Gyagenda & Engelhard, 1998a;Gyagenda & Engelhard, 1998b;Hughes, Keeling & Tuck, 1980;Hughes, Keeling & Tuck, 1983;Hughes & Keeling, 1984;Kan, 2005;Klein & Hart, 1968;Klein & Taub, 2005;Marshall & Powers, 1969;Murphy & Balzer, 1989;Schaefer, 2008;Slomp, 2012;Sulsky & Balzer, 1988;Wexley & Youtz, 1985;Woehr & Huffcutt, 1994). Fisher, Brooks, and Lewis (2002) state fitness for purpose requirement is the core of all testing work, and direct writing assessments are subjective and thereby more prone to reliability issues.…”

Rubrics in the classroom: do teachers really follow them?

“…In the rating process, various factors come into play: rater characteristics towards severity or leniency (Schaefer, 2008;Shi 2001), rater training experience (Huot, 1990;Weigle, 1998Weigle, , 2002), rater's language background (Kondo-Brown, 2002;Lumley & McNamara, 1995), and task variability (O'Loughlin and Wigglesworth 2007) are factors that have been researched over the years in performance assessment. Past research on rubric studies has focused on investigating changes in rater reliability (Lumley & McNamara 1995;McNamara, 1996;Weigle, 1998).…”