Examining Rater Errors in the Assessment of Written Composition With a Many‐Faceted Rasch Model

Abstract

This study describes several categories of rater errors (rater severity, halo effect, central tendency, and restriction of range). Criteria are presented for evaluating the quality of ratings based on a many‐faceted Rasch measurement (FACETS) model for analyzing judgments. A random sample of 264 compositions rated by 15 raters and a validity committee from the 1990 administration of the Eighth Grade Writing Test in Georgia is used to illustrate the model. The data suggest that there are significant differences in rater severity. Evidence of a halo effect is found for two raters who appear to be rating the compositions holistically rather than analytically. Approximately 80% of the ratings are in the two middle categories of the rating scale, indicating that the error of central tendency is present. Restriction of range is evident when the unadjusted raw score distribution is examined, although this rater error is less evident when adjusted estimates of writing competence are used

Psychometric evaluation of a new assessment of the ability to manage technology in everyday life

“…Raters' acceptable goodness-of-fit was set as outfit MnSq ˃ 0.6 and ˂1.5 (36). It is commonly accepted that 5% of the responses (i.e., person, technology, performance skill item and rater) are expected to be misfits by chance with z-values less than 2.…”

Measuring Essay Assessment: Intra-rater and Inter-rater Reliability

“…Assessing writing ability and the reliability of ratings have been a challenging concern for decades and there is always variation in the elements of writing preferred by raters and there are extraneous factors causing variation (Blok, 1985;Chase, 1968;Chase, 1983;Darus, 2006;East, 2009;Engelhard, 1994;Gyagenda & Engelhard, 1998a;Gyagenda & Engelhard, 1998b;Hughes, Keeling & Tuck, 1980;Hughes, Keeling & Tuck, 1983;Hughes & Keeling, 1984;Kan, 2005;Klein & Hart, 1968;Klein & Taub, 2005;Marshall & Powers, 1969;Murphy & Balzer, 1989;Schaefer, 2008;Slomp, 2012;Sulsky & Balzer, 1988;Wexley & Youtz, 1985;Woehr & Huffcutt, 1994). Fisher, Brooks, and Lewis (2002) state fitness for purpose requirement is the core of all testing work, and direct writing assessments are subjective and thereby more prone to reliability issues.…”

Involvement of Learners’ Characteristics within the Allocation of Submissions in the context of Peer Assessment in MOOCs

“…Some other models use this measurement as the multifaceted Rasch measurement model (FACETS) proposed by Linacre (1989) consists of three facets: the student competencies, the difficulty of the field or the assignment, and the differences between the severity of assessors [38].…”