The effect of training and rater differences on oral proficiency assessment

Abstract

As a result of the fact that judgments of non-native speech are closely tied to social biases, oral proficiency ratings are susceptible to error because of rater background and social attitudes. In the present study we seek first to estimate the variance attributable to rater background and attitudinal variables on novice raters’ assessments of L2 spoken English. Second, we examine the effects of minimal training in reducing the potency of those trait-irrelevant rater factors. Accordingly, we examined the relative impact of rater differences on TOEFL iBT® speaking scores. Eighty-two untrained raters judged 112 speech samples produced by TOEFL® examinees. Findings revealed that approximately 20% of untrained raters’ score variance was, in part, a result of their background and attitudinal factors. The strongest predictor was the raters’ own native speaker status. However, minimal online training dramatically reduced the impact of rater background and attitudinal variables for a subsample of high- and low-severity raters. Implications suggest that brief and user-friendly rater-training sessions offer the promise of mitigating rater bias, at least in the short run. This procedure can be adopted in assessment and other related fields of applied linguistics.

Listener Background in L2 Speech Evaluation

“…Current research is equivocal on the effect of listeners' linguistic knowledge and educational accomplishment on the assessment of L2 performance. The findings of the study conducted by Kang et al [3] did not reveal any significant relation between the degree of educational accomplishment and linguistic knowledge on listeners' holistic ratings of L2 proficiency. Also, listeners varying in degrees of educational accomplishment did not differ in rating severity; nevertheless, those listeners who were extremely severe or lenient in their assessments, regardless of their educational background, became more moderate after they received training and became familiar with assessment criteria.…”

“…Hsieh reported that English as a second language (ESL) teachers were more lenient in their ratings of ITAs' oral proficiency, comprehensibility, and accentedness than American undergraduates with no teaching experience. However, in a more recent study, Kang et al [3] did not find a statistically significant relationship between listeners' amount of teaching experience and their holistic ratings as well as their rating severity of L2 proficiency. Thus, although listeners who possess language teaching experience are assumed to be more lenient than their counterparts with no experience in teaching, this leniency is not consistent across the board.…”

“…The members of this community tend to be rated less favorably on the indices of status and solidarity by NSs [2]. In Kang et al [3] study, RLS emerged as one of the listener-related variables which significantly predicted naïve listeners' deviation from expert scores, implying that those who had the propensity to become engaged in RLS were more likely to be influenced by subjective impressions and provide less accurate ratings. Overall, listeners' perception can be influenced by a variety of nonlinguistic or paralinguistic aspects of an utterance.…”

“…On the one hand, some studies reported that NNSs tend to be more stringent in their assessments than NSs [5][6][7]. This severity could be justified in the light of the fact that "NNSs may reflect their personal L2 learning struggles, leading them to attribute examinees' errors to a lack of language learning effort on the part of the test takers" ( [3], p. 5). On the other hand, some other studies reported contradictory findings in which NSs were more severe listeners than NNSs [8].…”

“…However, human raters are vulnerable to be impacted by listeners' bias. Listeners who harbor negative expectations toward a certain group of speakers due to their social group affiliations, nationality, and un-prestigious accent have the proclivity to assess accented speech more negatively [1][2][3]. The bias formed mainly by listeners' background factors, such as listeners' predispositions, attitudes, expectations, and stereotypes, compromises the validity of oral proficiency assessment and unrightfully contributes to the speakers' oral proficiency score variance.…”