Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?

Zhang, Ying; Elder, Catherine

doi:10.1177/0265532209360671

Cited by 83 publications

(29 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the NS-NNS dichotomy has been questioned by World Englishes scholars (e.g., Davies, 1999). Furthermore, previous studies have found that raters' scores did not always reflect how raters interpreted and applied the scoring rubrics (e.g., Zhang & Elder, 2011). To address these concerns, this study used Kachru's framework of three concentric circles (1982,1985) to categorize rater types and a mixed-method design to investigate the qualitative differences in scoring processes and the quantitative differences in scores.…”

Section: Introductionmentioning

confidence: 99%

“…Kim (2009) compared the differences in Korean English teachers and Canadian English teachers in judging Korean English learners' oral proficiency. Zhang and Elder (2011) compared Chinese native speakers and English native speakers' ratings of Chinese students' oral proficiency in the national College English Test-Spoken English Test (CET-SET) in China. More recently, however, researchers have become more interested in speakers of English varieties from Outer Circle countries such as India (Carey et al, 2011;Hsu, 2012;Xi & Mollaun, 2011).…”

Section: Introductionmentioning

confidence: 99%

“…First, quantitative analyses of individual rater differences have mainly been conducted by using Multifaceted Rasch Analysis (MFRA) on harshness, consistency, and perception of item difficulty (Brown, 1995;Kim, 2009;Winke, Gass, & Myford, 2013;Zhang & Elder, 2011). Then group differences have been explored by using t-tests (Brown, 1995;Fayer & Krasinski, 1987;Winke et al, 2013), chi-square statistics (Brown, 1995;Zhang & Elder, 2011), or Multivariate Analysis of Variance (MANOVA) (Chalhoub-Deville & Wigglesworth, 2005). Other techniques, such as G-theory and logistic regression analyses, were also used by previous studies (Carey et al, 2011;Xi & Mollaun, 2011).…”

Section: Introductionmentioning

confidence: 99%

“…Outcomes of these studies have been mixed: Studies that compared NS with NNS groups showed that NSs generally rated as well as NNSs (i.e., either on a similar level of consistency and harshness or similar reliability), particularly for holistic scores (e.g., Xi & Mollaun, 2011;Zhang & Elder, 2011). Even in the cases where differences in harshness were flagged in the mean statistic, inferential tests often indicated that such mean differences were not statistically significant (Brown, 1995;Fayer & Krasinski, 1987).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Investigating Differences Between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks

Wei

Llosa

2015

Language Assessment Quarterly

View full text Add to dashboard Cite

This article reports on an investigation of the role raters' language background plays in raters' assessment of test takers' speaking ability. Specifically, this article examines differences between American and Indian raters in their scores and scoring processes when rating Indian test takers' responses to the Test of English as a Foreign Language TM Internet-Based Test (TOEFL iBT ® ) Speaking tasks. Three American and three Indian raters were asked to score 60 speech samples from 10 Indian test takers' responses to TOEFL iBT Speaking tasks and to perform think-aloud protocols while scoring. The data were analyzed with Multifaceted Rasch and verbal protocol analyses. Findings indicate that Indian raters were better than American raters at identifying and understanding features of Indian English in the test takers' responses. However, Indian and American raters did not differ in their use of scoring criteria, their attitudes toward Indian English, or in the internal consistency and severity of the scores.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Investigating Differences Between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks

Wei

Llosa

2015

Language Assessment Quarterly

View full text Add to dashboard Cite

show abstract

“…For example, Zhang and Elder (2011) in their study report that native and non-native English speaking teachers weigh various features of oral proficiency in making their decisions on Input source and EFL learners' perceptions towards it, on their listening performances, across gender…”

Section: Introductionmentioning

confidence: 99%

The effect of input source (Native vs. Non- native) and EFL learners’ perceptions towards it, on their listening performances, across gender

Sadeghi¹,

Zeinali²

2013

IJRSLL

View full text Add to dashboard Cite

The issue of the "native/non-native speaker's input source" occupies an important place in any EFL listening comprehension test. This study is meant to unveil whether listening to a native or non-native speaker (i.e., input source) and students' perceptions towards it affect the performance of upper-intermediate EFL learners in a listening test. For this purpose, an experimental design was used to compare the performance of two groups of learners on an EFL listening test. A test of 20 multiple choice items was administered to 66 EFL learners (31 male and 35 female), half of whom listened to a native speaker's voice while the other 33 test takers listened to a non-native speaker's voice. Moreover, a perception questionnaire considering students' perceptions towards using native or non-native input source in listening tests was utilized. The results of the study highlighted that the overall performance of the two groups differed significantly. That is, the listeners to the non-native input outperformed those who listened to a native speaker. Also, the results of the questionnaire revealed some noteworthy findings which indicate that students preferred the use of non-native input in listening tests. Moreover, considering gender as a moderator variable, a statistically significant main effect was found for gender; that is, gender does play a significant role as a moderator variable. In other words, female test takers performed better than males in both conditions. Further findings and implications are discussed in the paper.

show abstract

Assessing Speaking

O’Sullivan

2013

The Companion to Language Assessment

View full text Add to dashboard Cite

While there have been some significant advances in our understanding of spoken language, in terms of both production and interaction, over the past decades, there remain a number of areas of significant concern to the test writer. These are most notably construct definition (what exactly we are assessing), predictability of task response (task description), the effect of characteristics of the test taker on performance, the effect of characteristics associated with the interlocutor (the person with whom the candidate is interacting) on performance, the appropriateness of the scoring system (i.e., rating scale validity and reliability, and the reliability of the rating process). Before exploring these concerns, it is important to acknowledge the central role in test development of validation by first making explicit the model of validation that underpins this chapter. Thus, the chapter outlines a view of testing speaking that is based on a model of test validation. The chapter also addresses issues in testing speaking such as the test taker (in terms of physical characteristics, psychological characteristics, experiential characteristics, and cognitive aspects), the test system (performance parameters such as live, recorded, and automated formats, linguistic demands, and test administration), and the scoring system (theoretical fit, accuracy of decisions, and value of decisions).

show abstract

Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?

Cited by 83 publications

References 55 publications

Investigating Differences Between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks

Investigating Differences Between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks

The effect of input source (Native vs. Non- native) and EFL learners’ perceptions towards it, on their listening performances, across gender

Assessing Speaking

Contact Info

Product

Resources

About