Multiple Data Augmentation Strategies for Improving Performance on Automatic Short Answer Scoring

Lun, Jiaqi; Zhu, Jingtao; Tang, Yong; Yang, Min

doi:10.1609/aaai.v34i09.7062

Cited by 45 publications

(21 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This tendency is consistent with previous Comparing the conventional DNN-AES models shows that the LSTM-based model with MoT pooling has higher performance than models with last pooling, which is also consistent with previous studies (Alikaniotis et al, 2016;Riordan et al, 2017). BERT tends to outperform the LSTM-based models, as in other BERT applications including automated short-answer grading (Devlin et al, 2019;Lun et al, 2020;Sung et al, 2019). As Dasgupta et al (2018) reported, the conventional hybrid model shows the highest average accuracy among the conventional models.…”

Section: Resultssupporting

confidence: 90%

Neural Automated Essay Scoring Incorporating Handcrafted Features

Uto¹,

Xie²,

Ueno³

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Automated essay scoring (AES) is the task of automatically assigning scores to essays as an alternative to grading by human raters. Conventional AES typically relies on handcrafted features, whereas recent studies have proposed AES models based on deep neural networks (DNNs) to obviate the need for feature engineering. Furthermore, hybrid methods that integrate handcrafted features in a DNN-AES model have been recently developed and have achieved state-of-the-art accuracy. One of the most popular hybrid methods is formulated as a DNN-AES model with an additional recurrent neural network (RNN) that processes a sequence of handcrafted sentencelevel features. However, this method has the following problems: 1) It cannot incorporate effective essay-level features developed in previous AES research. 2) It greatly increases the numbers of model parameters and tuning parameters, increasing the difficulty of model training. 3) It has an additional RNN to process sentence-level features, enabling extension to various DNN-AES models complex. To resolve these problems, we propose a new hybrid method that integrates handcrafted essay-level features into a DNN-AES model. Specifically, our method concatenates handcrafted essay-level features to a distributed essay representation vector, which is obtained from an intermediate layer of a DNN-AES model. Our method is a simple DNN-AES extension, but significantly improves scoring accuracy.

show abstract

Section: Resultssupporting

confidence: 90%

Neural Automated Essay Scoring Incorporating Handcrafted Features

Uto¹,

Xie²,

Ueno³

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Jiaqi Lun et al (2020) proposed an automatic short answer scoring with BERT. In this with a reference answer comparing student responses and assigning scores.…”

Section: Neural Network Modelsmentioning

confidence: 99%

An automated essay scoring systems: a systematic literature review

2021

View full text Add to dashboard Cite

Assessment in the Education system plays a significant role in judging student performance. The present evaluation system is through human assessment. As the number of teachers' student ratio is gradually increasing, the manual evaluation process becomes complicated. The drawback of manual evaluation is that it is time-consuming, lacks reliability, and many more. This connection online examination system evolved as an alternative tool for pen and paper-based methods. Present Computer-based evaluation system works only for multiple-choice questions, but there is no proper evaluation system for grading essays and short answers. Many researchers are working on automated essay grading and short answer scoring for the last few decades, but assessing an essay by considering all parameters like the relevance of the content to the prompt, development of ideas, Cohesion, and Coherence is a big challenge till now. Few researchers focused on Content-based evaluation, while many of them addressed style-based assessment. This paper provides a systematic literature review on automated essay scoring systems. We studied the Artificial Intelligence and Machine Learning techniques used to evaluate automatic essay scoring and analyzed the limitations of the current studies and research trends. We observed that the essay evaluation is not done based on the relevance of the content and coherence.

show abstract

“…Additionally, some approaches consider the question (Lv et al, 2021), student models (Zhang et al, 2020b) or results from True/False questions posed in the same assessment (Uto and Uchida, 2020). Transformer-based approaches are also noteworthy here (Sung et al, 2019;Ghavidel et al, 2020;Lun et al, 2020;Camus and Filighera, 2020). They achieve high performances on the SemEval short answer grading benchmark dataset (Dzikovska et al, 2013).…”

Section: Automatic Short Answer Gradingmentioning

confidence: 99%

Cheating Automatic Short Answer Grading: On the Adversarial Usage of Adjectives and Adverbs

Filighera¹,

Ochs²,

Steuer³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatic grading models are valued for the time and effort saved during the instruction of large student bodies. Especially with the increasing digitization of education and interest in large-scale standardized testing, the popularity of automatic grading has risen to the point where commercial solutions are widely available and used. However, for short answer formats, automatic grading is challenging due to natural language ambiguity and versatility. While automatic short answer grading models are beginning to compare to human performance on some datasets, their robustness, especially to adversarially manipulated data, is questionable. Exploitable vulnerabilities in grading models can have far-reaching consequences ranging from cheating students receiving undeserved credit to undermining automatic grading altogether -even when most predictions are valid. In this paper, we devise a black-box adversarial attack tailored to the educational short answer grading scenario to investigate the grading models' robustness. In our attack, we insert adjectives and adverbs into natural places of incorrect student answers, fooling the model into predicting them as correct. We observed a loss of prediction accuracy between 10 and 22 percentage points using the state-of-the-art models BERT and T5. While our attack made answers appear less natural to humans

show abstract

Multiple Data Augmentation Strategies for Improving Performance on Automatic Short Answer Scoring

Cited by 45 publications

References 9 publications

Neural Automated Essay Scoring Incorporating Handcrafted Features

Neural Automated Essay Scoring Incorporating Handcrafted Features

An automated essay scoring systems: a systematic literature review

Cheating Automatic Short Answer Grading: On the Adversarial Usage of Adjectives and Adverbs

Contact Info

Product

Resources

About