Raters’ Fatigue and Their Comments During Scoring Writing Essays: A Case of Iranian Efl Learners

Abstract

Rating accuracy in writing among EFL learners is crucial in determining their English proficiency. Despite the importance of its accuracy, little is known about various factors may affect the accuracy of rating writing essays. This study examines how raters' comments on EFL writing tasks change as a result of fatigue. To this end, four raters were selected and each given 28 essays to score and comment on. Six general types of raters' comments (i.e., those on grammar, choice of words, organization, punctuation, dictation, and capitalization) were into focus in this study. Overall, results suggested that fatigue affects raters' frequency of comments on grammar, choice of words, and organization, and that raters' comments on punctuation, dictation, and capitalization do not seem to change significantly due to the effect of fatigue. Furthermore, this study revealed that the most and least frequent comments in 112 scored essays were those on grammar and dictation, respectively.

Building an Initial Validity Argument for Binary and Analytic Rating Scales for an EFL Classroom Writing Assessment: Evidence from Many-Facets Rasch Measurement

“…The high severity variability and low interrater agreement in this research imply that despite receiving the rater training, the raters still differed in interpreting the criteria, probably resulting from insufficient rater training and/or unclear rating criteria. Other factors that might have influenced the rater variability in this study could be those found in previous research, including rater fatigue (Mahshanian et al, 2017), rater personality (Wiseman, 2012), rater perceptions of criterion importance (Eckes, 2012), rater rating styles, strategies and preferences (Han, 2017), and essay characteristics (Han, 2017;Khamboonruang, 2020;Şahan & Razı, 2020). The high severity variability in this study supports a body of research which revealed that even experienced and well-trained raters still differed in severity (Khamboonruang, 2020;Li, 2022;Mendoza & Knoch, 2018;Yan & Chuang, 2022).…”

“…Some raters viewed a binary scale as easy to judge and practical (Jeong, 2019;Khamboonruang, 2020;Park & Yan, 2019) and some perceived that a binary scale was cognitive-loaded and difficult to judge (Kim, 2010;Park & Yan, 2019). In terms of rater behaviours, previous research has discovered that raters' rating variability was influenced by training (Yan & Chuang, 2022), time of rating (Lamprianou et al, 2021), writing genres (Jeong, 2017;Jiuliang, 2014), and rater characteristics, including but not limited to rater experience (Barkaoui, 2010(Barkaoui, , 2011Şahan & Razı, 2020), rater fatigue (Mahshanian et al, 2017), rater personality (Zhu et al, 2021), rater age (Isbell, 2017), raters perceptions of criterion importance (Eckes, 2012), and rater styles, strategies and preferences (Han, 2017). Prior studies also found that raters were more consistent in rating higher-quality essays than poorer-quality essays (Han, 2017;Khamboonruang, 2020;Şahan & Razı, 2020), and still significantly differed in their levels of severity even though well-trained and experienced (e.g., Khamboonruang, 2020;Li, 2022;Mendoza & Knoch, 2018;Yan & Chuang, 2022).…”

Target Needs in Writing Essay Material Based on Brainwriting Strategy

“…This is in line with the research conducted by Amir which states that the problems that occur in essay writing skills are due to a sense of self-distrust and feeling bored will express ideas or ideas on writing. [3] After getting some information about the gaps that occur in writing English essay in the field, the author was inspired to develop teaching material for writing essays compiled with one of the strategies devoted to writing skills, namely brainwriting strategies.…”