Automated Essay Scoring and the Deep Learning Black Box: How Are Rubric Scores Determined?

Kumar, Vive; Boulanger, David

doi:10.1007/s40593-020-00211-5

Cited by 38 publications

(35 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As previously mentioned, this paper reuses the AES system developed by Kumar and Boulanger (2020). The AES models were trained using the ASAP's seventh essay corpus.…”

Section: Automated Essay Scoring System Dataset and Feature Selectionmentioning

confidence: 99%

“…Rubric scores were resolved by adding the rubric scores assigned by the two human raters, producing a resolved rubric score between 0 and 6. This paper is a continuation of Boulanger and Kumar (2018, 2019 and Kumar and Boulanger (2020) where the objective is to open the AES black box to explain the holistic and rubric scores that it predicts. Essentially, the holistic score Kumar, 2018, 2019) is determined and justified through its four rubrics.…”

Section: Automated Essay Scoring System Dataset and Feature Selectionmentioning

confidence: 99%

“…Automated essay scoring (AES) is a compelling topic in Learning Analytics (LA) for the primary reason that recent advances in AI find it as a good testbed to explore artificial supplementation of human creativity. However, a vast swath of research tackles AES only holistically; only a few have even developed AES models at the rubric level, the very first layer of explanation underlying the prediction of holistic scores (Kumar et al, 2017;Taghipour, 2017;Kumar and Boulanger, 2020). None has attempted to explain the whole decision process of AES, from holistic scores to rubric scores and from rubric scores to writing feature modeling.…”

Section: Introductionmentioning

confidence: 99%

“…Related to this study and the work of Kumar and Boulanger (2020) is Revision Assistant, a commercial AES system developed by Turnitin (Woods et al, 2017;West-Smith et al, 2018), which in addition to predicting essays' holistic scores provides formative, rubric-specific, and sentence-level feedback over multiple drafts of a student's essay. The implementation of Revision Assistant moved away from the traditional approach to AES, which consists in using a limited set of features engineered by human experts representing only high-level characteristics of essays.…”

Section: Introductionmentioning

confidence: 99%

“…Again, except for Revision Assistant which undertook a holistic approach to AES including holistic and rubric scoring and provision of rubric-specific feedback at the sentence level, AES has generally not been investigated as a whole or as an endto-end product. Hence, the AES used in this study and developed by Kumar and Boulanger (2020) is unique in that it uses both deep learning (multi-layer perceptron neural network) and a huge pool of linguistic indices (1592), predicts both holistic and rubric scores, explaining holistic scores in terms of rubric scores, and reports which linguistic indices are the most important by rubric. This study, however, goes one step further and showcases how to explain the decision process behind the prediction of a rubric score for a specific essay, one of the main AES limitations identified in the literature (Taghipour, 2017) that this research intends to address, at least partially.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Explainable Automated Essay Scoring: Deep Learning Really Has Pedagogical Value

Kumar

Boulanger

2020

Front. Educ.

Self Cite

View full text Add to dashboard Cite

Automated essay scoring (AES) is a compelling topic in Learning Analytics for the primary reason that recent advances in AI find it as a good testbed to explore artificial supplementation of human creativity. However, a vast swath of research tackles AES only holistically; few have even developed AES models at the rubric level, the very first layer of explanation underlying the prediction of holistic scores. Consequently, the AES black box has remained impenetrable. Although several algorithms from Explainable Artificial Intelligence have recently been published, no research has yet investigated the role that these explanation models can play in: (a) discovering the decision-making process that drives AES, (b) fine-tuning predictive models to improve generalizability and interpretability, and (c) providing personalized, formative, and fine-grained feedback to students during the writing process. Building on previous studies where models were trained to predict both the holistic and rubric scores of essays, using the Automated Student Assessment Prize's essay datasets, this study focuses on predicting the quality of the writing style of Grade-7 essays and exposes the decision processes that lead to these predictions. In doing so, it evaluates the impact of deep learning (multilayer perceptron neural networks) on the performance of AES. It has been found that the effect of deep learning can be best viewed when assessing the trustworthiness of explanation models. As more hidden layers were added to the neural network, the descriptive accuracy increased by about 10%. This study shows that faster (up to three orders of magnitude) SHAP implementations are as accurate as the slower model-agnostic one. It leverages the state-of-the-art in natural language processing, applying feature selection on a pool of 1592 linguistic indices that measure aspects of text cohesion, lexical diversity, lexical sophistication, and syntactic sophistication and complexity. In addition to the list of most globally important features, this study reports (a) a list of features that are important for a specific essay (locally), (b) a range of values for each feature that contribute to higher or lower rubric scores, and (c) a model that allows to quantify the impact of the implementation of formative feedback.

show abstract

“…As previously mentioned, this paper reuses the AES system developed by Kumar and Boulanger (2020). The AES models were trained using the ASAP's seventh essay corpus.…”

Section: Automated Essay Scoring System Dataset and Feature Selectionmentioning

confidence: 99%