Rater reliability in language assessment: the bug of all bears

Gamaroff, Raphael

doi:10.1016/s0346-251x(99)00059-7

Cited by 25 publications

(14 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Essentially, the rhetorical, ideational, and language foci correspond closely to weighting patterns. In qualitative studies on weighting patterns similar to Cumming (1990), comments elicited from raters are systematically coded and frequencies of different comments are compared to infer the relevant weights attached to different scoring criteria, where a higher frequency is associated with a heavier weight (Gamaroff, 2000;Vaughan, 1991).…”

Section: Raters' Weighting Patternsmentioning

confidence: 99%

“…While early studies (e.g., Cumming, 1990;Gamaroff, 2000;Vaughan, 1991) normally treated raters as a homogeneous group, more recent studies on writing assessment started to address systematic differences between raters in weighting patterns. For example, Erdosy (2004) explored how four raters of the Test of English as a Foreign Language 4 (TOEFL) essays constructed scoring criteria without a scoring rubric, with a focus on the frequencies of two general categories: rhetorical-ideational focus and language focus.…”

Section: Rater Classificationmentioning

confidence: 99%

See 1 more Smart Citation

Weight-Based Classification of Raters and Rater Cognition in an EFL Speaking Test

Cai

2015

Language Assessment Quarterly

View full text Add to dashboard Cite

This study is an attempt to classify raters according to their weighting patterns and explore systematic differences between rater types in the rating process. In the context of an EFL speaking test, 126 raters were classified into three types-form-oriented, balanced, and content-oriented-through cluster analyses of their weighting patterns derived from a holistic judgment task. In subsequent verbal protocols, a smaller sample of raters from each type rated the performance of real test takers and justified their ratings. The procedure for classifying raters yielded reliable results that agreed broadly with self-perceived and actual weights of the raters in the rating process. Verbal protocol analyses found systematic differences across rater types in the distinction between criteria and conformity to rubric-implied criteria. The form-oriented raters were found to overweigh form-related criteria and underweigh content-related criteria, which suggested unbalanced construct representation and led to more halos. Preliminary evidence also suggested that the form-oriented raters might overrate digressive content due to lower sensitivity to digression.

show abstract

Section: Raters' Weighting Patternsmentioning

confidence: 99%

Section: Rater Classificationmentioning

confidence: 99%

Weight-Based Classification of Raters and Rater Cognition in an EFL Speaking Test

Cai

2015

Language Assessment Quarterly

View full text Add to dashboard Cite

show abstract

“…Una de las críticas más frecuentes a los estudios que intentan medir las habilidades de escritura tiene relación con la confiabilidad de las evaluaciones, pues el lenguaje es siempre objeto de interpretación. Aparte de la calidad de la escritura, múltiples fuentes de error pueden contribuir a la variabilidad de los puntajes, entre ellas, diferencias de criterio entre correctores, ambigüedad de los criterios de corrección y variaciones de las condiciones en que ésta se realiza [15][16][17] .…”

Section: N V E S T I G a C I ó Nunclassified

Mejoría en las habilidades de comunicación escrita de estudiantes de medicina: Impacto de un taller de escritura

et al. 2009

View full text Add to dashboard Cite

show abstract

“…However, fluctuations in scores associated with rater factors are extensive (Huot 1990;Lumley and McNamara 1995;Weigle 1998;Gamaroff 2000;KondoBrown 2002;Amengual 2003;2004). Furthermore, raters are recognised to be one of the main sources of measurement error in assessing a candidate's performance (Milanovic et al 1996;Herrera 2001).…”

Section: Introductionmentioning

confidence: 99%

“…Since trying to reconcile raters' subjectivity with objective precision seems extremely difficult to achieve, rater reliability has been defined as the greatest bugbear in assessment (Moss 1994;Gamaroff 2000).…”

Section: Introductionmentioning

confidence: 99%

Rater discrepancy in the spanish university entrance examination

Pizarro

2004

J. engl. stud. (Logroño, internet)

View full text Add to dashboard Cite

ABSTRACT. This study investigated the level of inter-rater and intra-rater reliability of thirty-two raters in the evaluation of the composition subtest of the English Test (ET) in the Spanish University Entrance Examination (SUEEIt is well established that some raters are consistently harsher in their assessment of a candidate's ability than others […] From the point of view of the candidate, it becomes a matter of luck whether they are assessed by particular raters. A candidate may draw the most lenient member of the rating team and benefit as a result or, alternatively, s/he may draw the harshest member and may suffer the consequences of this. (Wigglesworth 1993: 305).

show abstract

Rater reliability in language assessment: the bug of all bears

Cited by 25 publications

References 11 publications

Weight-Based Classification of Raters and Rater Cognition in an EFL Speaking Test

Weight-Based Classification of Raters and Rater Cognition in an EFL Speaking Test

Mejoría en las habilidades de comunicación escrita de estudiantes de medicina: Impacto de un taller de escritura

Rater discrepancy in the spanish university entrance examination

Contact Info

Product

Resources

About