An examination of rater performance on a local oral English proficiency test: A mixed-methods approach

Abstract

This paper reports on a mixed-methods approach to evaluate rater performance on a local oral English proficiency test. Three types of reliability estimates were reported to examine rater performance from different perspectives. Quantitative results were also triangulated with qualitative rater comments to arrive at a more representative picture of rater performance and to inform rater training. Specifically, both quantitative (6338 valid rating scores) and qualitative data (506 sets of rater comments) were analyzed with respect to rater consistency, rater consensus, rater severity, rater interaction, and raters’ use of rating scale. While raters achieved overall satisfactory inter-rater reliability (r = .73), they differed in severity and achieved relatively low exact score agreement. Disagreement of rating scores was largely explained by two significant main effects: (1) examinees’ oral English proficiency level, that is, raters tend to agree more on higher score levels than on lower score levels; (2) raters’ differential severity due to raters’ varied perceptions of speech intelligibility toward Indian and low-proficient Chinese examinees. However, effect sizes of raters’ differential severity effect on overall rater agreement were rather small, suggesting that varied perceptions among trained raters of second language (L2) intelligibility, though possible, are not likely to have a large impact on the overall evaluation of oral English proficiency. In contrast, at the lower score levels, examinees’ varied language proficiency profiles generated difficulty for rater alignment. Rater disagreement at these levels accounted for most of the overall rater disagreement and thus should be focused on during rater training. Implication of this study is that interpretation of rater performance should not just focus on identifying interactions between raters’ and examinees’ linguistic background but also examine the impact of rater interactions across examinees’ language proficiency levels. Findings of this study also indicate effectiveness of triangulating different sources of data on rater performance using a mixed-methods approach, especially in local testing contexts.

Cultural Effects on English Language Teachers’ Judgments of Metaphoric Creativity: A Mixed‐Methods Approach

“…Table also reports the outfit and infit indexes, which concern how well raters’ ratings fit the Rasch model or can be predicted by it (Bachman, ; Bond & Fox, ; Eckes, ). In assessing infit mean square and outfit mean square, although both dimensions are ideally expected to be 1.00 (Bond & Fox, ), Linacre () suggests reasonable fit can fall in between .50 and 1.50, which is widely applied in previous studies (e.g., He et al., ; Saeidi et al., ; Yan, ) as the criterion for evaluating both fit statistics.…”

“…Vertical rulers for rater severity and metaphoric topic. fall in between .50 and 1.50, which is widely applied in previous studies (e.g., He et al, 2013;Saeidi et al, 2013;Yan, 2014) as the criterion for evaluating both fit statistics.…”

Second Language Pronunciation Assessment

“…As was the case with writing, research has examined rater bias towards certain scale criteria and groups of test takers from the same language background (e.g. Yan, 2014), without examining possible causes of such biases. Other studies have focused on examining specific rater background variables and have investigated whether these can be attributed to certain biases in a dataset.…”

Calificación del test oral EEE-4: análisis del discurso basado basado en redes complejas

“…Sin embargo, por la pluralidad y diversidad de cada examinando individual una sola referencia no puede representar todas las características de sus expresiones orales. En las últimas investigaciones sobre el test oral, se presta una atención cada vez mayor a la coincidencia entre las calificaciones de cada uno de los evaluadores (Kim 2009;Yan 2014). En este trabajo llevamos a cabo un análisis del discurso mediante redes complejas sobre el test oral del examen EEE-4 (Examen del Español como Especialidad -Nivel 4), intentando encontrar una nueva metodología para ayudar a calificar las expresiones orales en diferentes niveles, lo que se propone como un apoyo potencial para los evaluadores.…”