Don't Just Judge the Spelling! The Influence of Spelling on Assessing Second-Language Student Essays

Abstract

When judging subject-specific aspects of students’ texts, teachers should assess various characteristics, e.g., spelling and content, independently of one another since these characteristics are indicators of different skills. Independent judgments enable teachers to adapt their classroom instruction according to students’ skills. It is still unclear how well teachers meet this challenge and which intervention could be helpful to them. In Study 1, N = 51 pre-service teachers assessed four authentic English as a Second Language (ESL) essays with different overall text qualities and different qualities of spelling using holistic and analytic rating scales. Results showed a negative influence of the experimentally manipulated spelling errors on the judgment of almost all textual characteristics. In Study 2, an experimental prompt was used to reduce this judgment error. Participants who were made aware of the judgment error caused by spelling errors formed their judgments in a less biased way, indicating a reduction of bias. The determinants of the observed effects and their practical implications are discussed.

Assessing the Quality of Argumentative Texts: Examining the General Agreement Between Different Rating Procedures and Exploring Inferences of (Dis)agreement Cases

“…Ideally, all viewpoints should be recognized and supported but the opposite viewpoint should be less convincing than the chosen viewpoint, as Stapleton and Wu (2015) declare. In the present study, we confirmed the results of several previous studies (Figueredo and Varnhagen, 2005;Barkaoui, 2010;De La Paz and Felton, 2010;Rezaei and Lovorn, 2010;Stapleton and Wu, 2015;Cuevas et al, 2016;Jansen et al, 2021;Syed et al, 2021). These studies showed that the elements that seem to be associated with strong texts were: (a) Use of the (adapted) Toulmin elements, (b) refuting all counterarguments by rebuttal(s), (c) integrating information from source texts, (d) strong persuasiveness, (e) factual accuracy, (f) use of introduction and conclusion, (g) long text length, and (h) good writing mechanics.…”

“…However, including unnecessary and irrelevant information in texts can hinder the flow and readability of a text. Finally, bad writing mechanics seem to negatively affect text quality (Figueredo and Varnhagen, 2005;Rezaei and Lovorn, 2010;Jansen et al, 2021). However, this list is not exhaustive.…”

Bias, Halo Effect and Horn Effect: A Systematic Literature Review

“…A total of four studies out of five studies focused on Bias and Halo effect (Jones & Bergin, 2019;Matlock et al, 2017;Sonnert et al, 2018;Sutu et al, 2019), and one study focused on the Halo effect only (Sheppard, 2019) in United States of Amerika (USA). Two studies out of three focused on Bias, Halo effect and Horn effect (Marono & Bartels, 2020;Xu et al, 2020), and only one study focus on bias and Halo effects (Brown et al, 2017) in the United Kingdom (UK), three studies focused on bias and Halo effect (Behrmann, 2019;Jansen et al, 2021;Nufer, 2019) in German, five countries focused on the exact keywords which are Bias and Halo effect in Brazil (Rego et al, 2018), Turkey (Yilmaz, 2017), Italy (Cantarelli et al, 2020), Iran (Bijani, 2018 and Korea (Gweon et al, 2017). Finally, one study concentrated on the Dutch's bias, Halo effect and Horn effect (Marucci et al, 2021) (Table 3).…”

“…A general impression of the teacher may overpower performance disparities in other categories (i.e., ratings), which may be highly correlated across categories (Sonnert et al, 2018). Consequently, students may be placed in suboptimal educational tracks based on biased scoring (Jansen et al, 2021). People may be clueless about their unconscious biases, so it is critical to warn them about their proclivity for bias in decision-making, as awareness of bias can help mitigate some of the negative consequences of decision-making (Ehrlinger et al, 2005) An initiative has been taken to minimise the raters' conflicting attitudes and biases that they might have brought to their assessment by giving them assessment criteria and performance samples and asking them to take part in the training program.…”

“…The halo effect's documentation in scenarios when an educator evaluates learners has yet to be investigated (Gweon et al, 2017). However, Jansen et al (2021) stated that spelling problems caused halo effects on analytic rating scales. These halo effects could be reduced by using a prompt.…”