Sequential Effects in Essay Ratings: Evidence of Assimilation Effects Using Cross-Classified Models

Abstract

Writing assessments are an indispensable part of most language competency tests. In our research, we used cross-classified models to study rater effects in the real essay rating process of a large-scale, high-stakes educational examination administered in China in 2011. Generally, four cross-classified models are suggested for investigation of rater effects: (1) the existence of sequential effects, (2) the direction of the sequential effects, and (3) differences in raters by their individual characteristics. We applied these models to the data to account for possible cluster effects caused by the application of multiple rating strategies. The results of our research showed that raters demonstrated sequential effects during the rating process. In contrast to many other studies on rater effects, our study found that raters exhibited assimilation effects. The more experienced, lenient, and qualified raters were less susceptible to assimilation effects. In addition, our research demonstrated the feasibility and appropriateness of using cross-classified models in assessing rater effects for such data structures. This paper also discusses the implications for educators and practitioners who are interested in reducing sequential effects in the rating process, and suggests directions for future research.

Scoring Stability in a Large‐Scale Assessment Program: A Longitudinal Analysis of Leniency/Severity Effects

“…For all item groups, the vast majority of the variance in score differences fell between validity responses within items. Previous research has shown scoring difficulty varies widely, depending on the nature of the response (Engelhard, 1996), and results align with a number of prior studies that have found individual responses to be one of the largest explainable sources of variation in rater accuracy (Baird et al, 2013;Leckie & Baird, 2011;Pinot de Moira, Massey, Baird, & Morrissey, 2002;Zhao et al, 2017).…”

“…Much attention has been given to rater effects such as leniency and severity (Myford & Wolfe, 2003;Saal, Downey, & Lahey, 1980;Wolfe, 2004). Raters have also been found to demonstrate halo (Ridge, 2001a(Ridge, , 2001bRudner, 1992), central tendency (Engelhard, 1994), and sequential effects (Attali, 2011;Zhao, Andersson, Guo, & Xin, 2017) during scoring. Researchers have examined the impact of rater background characteristics, such as demographics and experience (Song et al, 2014), and rater training methods (Raczynski, Cohen, Engelhard & Lu, 2015;Wolfe, Matthews, & Vickers, 2010) on scoring accuracy.…”

Sequential effects in facial attractiveness judgments using cross-classified models: Investigating perceptual and response biases.

“…This nonindependence is applicable because responses given by different participants to the same image tend to be similar. Linear mixed-effects models have now been applied to other types of sequential decisions (Zhao et al, 2017) but have yet to be employed by attractiveness researchers.…”

Psychophysical laws as reflection of mental space properties

“…Expression (40) for the capacity of direct quantitative comparison of images enables me, first, to justify Teghtsoonian's estimate of the dynamic range R M of sensory magnitude common for many perceptual continua, log R M ∼ 1.5 (Sec. 3, Proposition 2).…”

“…Judgments on perceived magnitudes are also affected by the recent history of evaluating physical stimuli belonging to the same perceptual continuum or, even, deferent one, which is often called sequential effects. Their various aspects were thoroughly studied in the last century and for a recent review a reader may be referred, e.g., to [37,38,39,40]. In particular, the sequential effects give rise to hysteresis in stimulus evaluation.…”