A Hierarchical Rater Model Approach for Integrating Automated Essay Scoring Models

Abstract

Essay writing tests, integral in many educational settings, demand significant resources for manual scoring. Automated essay scoring (AES) can alleviate this by automating the process, thereby reducing human effort. However, the multitude of AES models, each varying in its features and scoring approaches, complicates selecting one optimal model, especially when evaluating diverse content-related aspects across multiple rating items. Therefore, we propose a hierarchical rater model-based approach to integrate predictions from multiple AES models, accounting for their distinct scoring behaviors. We investigated its performance on data from a university essay writing test. The proposed method achieved accuracy that was comparable to the best individual AES model. This is a promising result because it additionally reduced the amount of differential item functioning between human and automated scoring and thus established a higher degree of measurement invariance compared to the individual AES models.

Natural Language Processing in Psychology

“…Finally, the seventh article (Fink et al, 2024) proposes a new approach to automated essay scoring that integrates the predictions of multiple scoring models. Specifically, a hierarchical rater model based on signal detection theory is applied to account for their different scoring behaviors.…”