Rater judgment and English language speaking proficiency

Abstract

The paper investigates whether there is a shared perception of speaking proficiency among raters from different English speaking countries. More specifically, this study examines whether there is a significant difference among English language learning (ELL) teachers, residing in Australia, Canada, the UK, and the USA when rating speech samples of international English language students. Teachers were asked to rate samples from international students who took the test of spoken English (TSE), the oral component of TOEFL. Building on previous research which has demonstrated that different tasks and rater groups affect results obtained from learner performance on oral tests, this project investigated both rating variation as a result of country of origin and variations due to TSE task effects. Multivariate analyses were used to analyze the ratings. Also effect sizes are reported.The present study investigates whether teachers from different English speaking countries, all of whom are native speakers, share similar perceptions in terms of their rating of the oral performance of English as a second language (ESL) test takers. The impetus behind this question is the ever-increasing efforts by major language testing organizations to market their tests in countries or with populations for which they were originally not intended and/or for which appropriate research has not been carried out to justify such wide use.This study is the first investigation in a research agenda that seeks to address issues regarding the comparability and characteristics of ESL speaking test ratings awarded by different international rater groups. In the present investigation, the focus is on the oral test of the Test of English as a Foreign Language (TOEFL) program, i.e., Test of Spoken English (TSE). (1995) argues that two organizations have dominated the large-scale English language testing industry throughout the world: the University of Cambridge ESOL Examinations (previously University of Cambridge Local Examinations Syndicate or UCLES) and the Educational Testing Service (ETS). These two organizations have different orientations, ideologies, and approaches to measuring non-native speakers' English language ability (see Chalhoub-Deville and Turner, 2000). For example, tests produced by the University of Cambridge ESOL Examinations have tended to be more achievement based with strong connections to English language teaching syllabi. On the other hand, ETS has distanced itself from any specific instructional program/textbook, * Professor of Foreign Language and ESL Education, LARGE-SCALE ENGLISH LANGUAGE TESTING Spolsky

Raters Knowledge of Students Proficiency Levels as a Source of Measurement Error in Oral Assessments

“…The findings of the study are in accordance with the literature which suggests that the constructirrelevant factors can influence the assessment of the raters and the scores of the test-takers in oral interviews (e.g., Chalhoub-Deville & Wigglesworth, 2005;Myford & Wolfe, 2000;O'Loughlin, 2002;O'Sullivan, 2000;Winke & Gass, 2012;Winke et al, 2011). Several factors that affect raters' scorings in oral interviews have been studied in the literature; however, to the knowledge of the researchers, no study has been conducted to investigate the effects of the raters' prior knowledge of the students' proficiency levels on their scoring behaviors during proficiency exams oral interviews.…”

“…In conclusion, the findings of the present study concur with the previous studies by confirming that raters may be affected by factors other than the actual performance of the test-takers (e.g., Chalhoub-Deville, 1995;Chalhoub-Deville & Wigglesworth, 2005;Lumley & McNamara, 1995;Myford & Wolfe, 2000;Winke & Gass, 2012). Whether random or systematic, similar to the other studies, measurement error was observed in this study underlining the influential factors that may cause disagreement within and/or among the raters' judgments in oral performance assessments.…”

“…Previous studies have investigated rater effects on oral test scores from different perspectives such as the raters' educational and professional experience (e.g., Chalhoub-Deville, 1995), raters' nationality and native language (e.g., Chalhoub-Deville & Wigglesworth, 2005;Winke & Gass, 2012;Winke et al, 2011), rater training (e.g., Lumley & McNamara, 1995;Myford & Wolfe, 2000), and the gender of candidates and/or interviewers (e.g., O'Loughlin, 2002;O'Sullivan, 2000). For instance, Lumley and McNamara (1995) examined the effect of rater training on the stability of rater characteristics and rater bias whereas MacIntyre, Noels, and Clément (1997) examined bias in self-ratings in terms of participants' perceived competence in an L2 in relation to their actual competence and language anxiety.…”

Investigating Differences Between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks

“…First, quantitative analyses of individual rater differences have mainly been conducted by using Multifaceted Rasch Analysis (MFRA) on harshness, consistency, and perception of item difficulty (Brown, 1995;Kim, 2009;Winke, Gass, & Myford, 2013;Zhang & Elder, 2011). Then group differences have been explored by using t-tests (Brown, 1995;Fayer & Krasinski, 1987;Winke et al, 2013), chi-square statistics (Brown, 1995;Zhang & Elder, 2011), or Multivariate Analysis of Variance (MANOVA) (Chalhoub-Deville & Wigglesworth, 2005). Other techniques, such as G-theory and logistic regression analyses, were also used by previous studies (Carey et al, 2011;Xi & Mollaun, 2011).…”

Rater attitude towards emerging varieties of English: a new rater effect?

“…Questions about bias on the part of raters towards the varieties of English in the world today are arising in the relevant discussions (Davies et al 2003). Studies have investigated the impact of rater nationality on speaking test scores (Chalhoub-Deville and Wigglesworth 2005;Hamp-Lyons and Zhang 2001), differences in scores due to rater attitudes towards Korean English (Kim 2005), and the recent development of a "rater attitude instrument" that measures raters' attitudes towards WE (Hsu 2016). The emerging agenda on rater psychological traits and attitude-behavior relationship includes broad concerns about the impact of WE on English speaking test scores, score validity (Davies et al 2003), fairness (Kunnan 2004), and unexpected consequences of test use (Davidson 2006).…”