Examining Human and Automated Ratings of Elementary Students’ Writing Quality: A Multivariate Generalizability Theory Application

Abstract

We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students ( n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3–5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.

Predictors of middle school students’ perceptions of automated writing evaluation

“…Hence, for the purposes of this study we measured students' writing proficiency using the Overall Score provided by PEG, which is the sum of the six trait scores and ranges from 6.0 to 30.0 points. The Overall Score is highly reliable (see Chen, Hebert, & Wilson, 2022 , Wilson, Chen, Sandbank, & Hebert, 2019 ).…”

A Latent Dirichlet Allocation approach to understanding students’ perceptions of Automated Writing Evaluation

“…The tool is designed to assist writing instruction and learning by offering immediate, automated feedback and scores in response to student writing [ 34 ]. At the time of the RCT, MI Write leveraged the automated essay scoring system Project Essay Grade (PEG; [ [35] , [36] , [37] ]). PEG provided students with an automated writing-quality score ranging from 6.0 to 30.0, as well as scores for six distinct writing traits, including idea development, organization, style, sentence fluency, word choice, and conventions, based on the Six Trait Model [ 38 ].…”

A Robust Indicator Mean-Based Method for Estimating Generalizability Theory Absolute Error and Related Dependability Indices within Structural Equation Modeling Frameworks

“…Generalizability theory (GT; [1][2][3][4]) offers an effective framework for accounting for multiple sources of measurement error when assessing the accuracy of measurement data and subsequently using that information to evaluate and improve assessment procedures. Such techniques have recently been applied to advantage in such diverse fields as education [5][6][7][8][9][10][11][12][13], psychology [14][15][16][17], business [18][19][20][21], medicine/health sciences [22][23][24][25][26][27][28][29], psychophysiology [30][31][32], athletic training [33][34][35], and many others. Although GT designs have traditionally been analyzed using analysis of variance (ANOVA) procedures, they also can be analyzed using linear mixed-effect [36,37] and structural equation models (SEMs; [14,[37][38][39][40][41][42][43]…”