Removing bias towards World Englishes: The development of a Rater Attitude Instrument using Indian English as a stimulus

Abstract

This study explores the attitudes of raters of English speaking tests towards the global spread of English and the challenges in rating speakers of Indian English in descriptive speaking tasks. The claims put forward by language attitude studies indicate a validity issue in English speaking tests: listeners tend to hold negative attitudes towards speakers of non-standard English, and judge them unfavorably. As there are no adequate measures of listener/rater attitude towards emerging varieties of English in language assessment research, a Rater Attitude Instrument comprising a three-phase self-measure was developed. It comprises 11 semantic differential scale items and 31 Likert scale items representing three attitude dimensions of feeling, cognition, and behavior tendency as claimed by psychologists. Confirmatory factor analysis supported a two-factor structure with acceptable model fit indices. This measure represents a new initiative to examine raters' psychological traits as a source of validity evidence in English speaking tests to strengthen arguments about test-takers' English language proficiency in response to the change of sociolinguistic landscape. The implications for norm selection in English oral tests are also discussed. Theoretical background World EnglishesNew lines of sociolinguistic research have acknowledged the pluricentricity of English such as world Englishes (WE) and English as lingua franca (ELF). The Kachru-led line of WE documents the function, status, linguistic maturity, and legitimacy of the emerging Unlike the constructs being conceptualized in the field of psychology, recent studies on rater attitude towards WE have revealed mixed and inconclusive findings. Kim (2005) examined the language backgrounds of raters, their attitudes toward WE, and how they scored the speech performance of six Korean students on the Test of Spoken English (TSE) picture description task, using holistic and analytic scales. Although their ratings on the holistic scales were fairly similar, their different attitudes towards WE significantly affected their analytic ratings on grammar, rate of speech, and task fulfillment, with those labeled as "positive" giving more lenient ratings.Chalhoub-Deville and Wigglesworth (2005) investigated the rating performance by raters from the inner circle countries, including Australia, Canada, the UK and the US, and found no significant difference in evaluating ESL test-takers' speaking performance.

Indonesian Pre-Service English Teachers’ Attitudes Towards the Use of World Englishes in International English Proficiency Tests: A Small-Scale Survey

“…The instrument of the present study was taken from Hamid (2014) and Hsu (2016) and modified by Monfared (2020) about participants' view on WE in English proficiency such as TOEFL and IELTS. The questionnaire used likert scale to measure the answer, 1strongly disagree; 2disagree; 3moderately disagree; 4moderately agree; 5agree; 6strongly agree.…”

Novice Teachers and Native-Speakerism: Identity Recognition on Teacher for English as An International Language

“…Studies have shown how native-speakerism is still concerning from both sides; the learners (Hendriks et al, 2021;Hsu, 2016;Lee & Warren Green, 2016;Tamimi Sa'd, 2018) and the instructors (Dogancay-Aktuna & Hardman, 2018). Hendriks et al (2021) share learners' attitudes toward their lecturers who teach with non-native accented speaking.…”

“…The findings show that the non-native accented lecturers were perceived negatively than lecturers with a native accent. Hsu (2016) conducted a study with the collaboration of English testing raters in India and found out that the raters more likely to value test-takers who speak in American or British English. Lee & Warren Green (2016) in South Korea state that it is easier to recognize native-accented speaking than any English variations.…”

Rater attitude towards emerging varieties of English: a new rater effect?

“…The rater attitude instrument (RAI) has been discussed in detail in Hsu (2016). This section will provide a brief summary of the RAI and highlight the aspects that are relevant to the current study.…”

“…Questions about bias on the part of raters towards the varieties of English in the world today are arising in the relevant discussions (Davies et al 2003). Studies have investigated the impact of rater nationality on speaking test scores (Chalhoub-Deville and Wigglesworth 2005;Hamp-Lyons and Zhang 2001), differences in scores due to rater attitudes towards Korean English (Kim 2005), and the recent development of a "rater attitude instrument" that measures raters' attitudes towards WE (Hsu 2016). The emerging agenda on rater psychological traits and attitude-behavior relationship includes broad concerns about the impact of WE on English speaking test scores, score validity (Davies et al 2003), fairness (Kunnan 2004), and unexpected consequences of test use (Davidson 2006).…”