Investigating the Effect of Training on Raters’ Bias toward Test Takers in Oral Proficiency Assessment : A FACETS Analysis

Abstract

Typically, variability among raters in scoring and their bias is mediated through rater training. However, questions still remain about whether training can affect raters' severity or leniency. Furthermore, few studies have looked at the differences between trained and untrained raters in oral assessment. Oral test scores of 200 test takers rated by 20 raters and were analyzed before and after a training program using the multifaceted Rasch measurement (MFRM). The results demonstrated the constructive impact of training programs in reducing raters' biases and increasing their consistency measures. This study indicated that inexperienced raters benefited more from a training program than experienced raters and thus achieved higher measures of consistency afterward. It also demonstrated a higher biased interaction for test takers on the extreme ends of the oral ability continuum. The findings demonstrated that it is almost impossible to completely eradicate rater variability even through rater training. Therefore, rater training should be viewed as a procedure to establish within-rater consistency rather than between-rater consistency. Since this study showed that inexperienced raters can rate even more reliably than experienced ones after training, there is no evidence whereby decision makers can exclude inexperienced raters solely because of their lack of adequate experience. Consequently, decision makers need to use their budgets for establishing rater training programs for inexperienced raters instead.

A Critical Review of IELTS Speaking Test

“…This level may not correspond to the candidate's inherent ability due to the fact that the IELTS Speaking test is administered by a single rater Karim & Haq, (2014). Test raters might be classified as either "the difficult interviewers" or "the easy interviews," according to Bijani, H., & Khabiri (2017). In particular, difficult interviewers place an emphasis on the complex abilities of speculating and defending ideas.…”

Performance-Based Assessment on Dispensing Drugs in English of Thai Pharmacy Students.

“…In like manner, a study by Bonk and Ockey (2003) found that the more experience the raters obtain, the higher their standards are raised. In addition, the experienced raters tend to have more bias than inexperienced raters even after the training (Bijani & Khabiri, 2017). However, the harsh rater in this study conformed to the group agreement and is not considered a threat to the assessment.…”

“…Similarly, utilizing the result from the MFRM to inform each rater's performance in assessing can lessen the raters' judgment effect (Bonk & Ockey, 2003). Based on previous studies (Bijani & Khabiri, 2017;Bonk & Ockey, 2003;Eckes, 2009), good training sessions should offer the raters the detail of their rating performance, which can be obtained from MFRM.…”