Creating Scoring Rubric from Representative Student Answers for Improved Short Answer Grading

Marvaniya, Smit; Saha, Swarnadeep; Dhamecha, Tejas I.; Foltz, Peter W.; Sindhgatta, Renuka; Sengupta, Bikram

doi:10.1145/3269206.3271755

Cited by 23 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Grading based on the similarity between student answers and one or more example answers has also been investigated by Marvaniya et al (2018) and Mohler et al (2011).…”

Section: Automated Gradingmentioning

confidence: 99%

Reducing Workload in Short Answer Grading Using Machine Learning

Weegar

Idestam-Almquist

2023

Int J Artif Intell Educ

View full text Add to dashboard Cite

Machine learning methods can be used to reduce the manual workload in exam grading, making it possible for teachers to spend more time on other tasks. However, when it comes to grading exams, fully eliminating manual work is not yet possible even with very accurate automated grading, as any grading mistakes could have significant consequences for the students. Here, the evaluation of an automated grading approach is therefore extended from measuring workload in relation to the accuracy of automated grading, to also measuring the overall workload required to correctly grade a full exam, with and without the support of machine learning. The evaluation was performed during an introductory computer science course with over 400 students. The exam consisted of 64 questions with relatively short answers and a two-step approach for automated grading was applied. First, a subset of answers to the exam questions was manually graded and next used as training data for machine learning models classifying the remaining answers. A number of different strategies for how to select which answers to include in the training data were evaluated. The time spent on different grading actions was measured along with the reduction of effort using clustering of answers and automated scoring. Compared to fully manual grading, the overall reduction of workload was substantial—between 64% and 74%—even with a complete manual review of all classifier output to ensure a fair grading.

show abstract

“…Grading based on the similarity between student answers and one or more example answers has also been investigated by Marvaniya et al (2018) and Mohler et al (2011).…”

Section: Automated Gradingmentioning

confidence: 99%

Reducing Workload in Short Answer Grading Using Machine Learning

Weegar

Idestam-Almquist

2023

Int J Artif Intell Educ

View full text Add to dashboard Cite

show abstract

“…In the early stages, deep learning was usually involved in ASAG tasks only as a complement to feature-based methods, where deep learning provides only some features, but not all, for the ASAG task. For example, Marvaniya et al [5] and Saha et al [2] used a pre-trained Bi-LSTM network InferSent [19] to obtain sentence embedding vectors of answer texts, which compensates for the disadvantage of not being able to represent the context in token overlap methods. Tan et al [20] proposed a scoring method using a combination of Graph Convolutional Networks (GCNs) with several sparse features.…”

Section: Related Studiesmentioning

confidence: 99%

“…𝑆 𝑐 = 𝑅𝑒𝐿𝑈(𝐶𝑁𝑁(𝑆)) ∈ ℝ 𝑑 𝐶 ×𝑛 (5) where 𝐶𝑁𝑁(⋅) denotes the Siamese CNN with zero padding and shared parameters, 𝑑 𝑐 is the number of convolution kernels, and 𝑅𝑒𝐿𝑈(⋅) denotes the Rectified Linear Unit activation function.…”

Section: Convolutional Layermentioning

confidence: 99%

“…Since then, many traditional machine learning methods using manual features have emerged. Examples include using token overlap to represent text similarity [2], [6], [7], [10]; using syntax and dependency to analyze and represent the answer text [3], [5]; using tf-idf [4], [7], [8], [11] to form the answer text vector; and utilizing the WordNet knowledge base [6], [7], edits [9] or sentence embedding [2], [5], [7] to computing text similarity. However, feature-based machine learning methods have many shortcomings.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Cross-Lingual Hybrid Neural Network With Interaction Enhancement for Grading Short-Answer Texts

Chen¹,

Luo²,

Zhu

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Automatic Short-Answer Grading (ASAG) is an application for recognizing textual entailment in smart education. With the continuous expansion of the application scope of artificial neural networks, many deep learning models have been applied to grading short-answer texts. However, the coding structures and interaction forms of existing models are still too simple to meet the requirements of the ASAG task, resulting in low scoring accuracy. To address these challenges, we propose a cross-lingual hybrid neural network with interaction enhancement for ASAG. First, we sequentially use a convolutional neural network and bidirectional Long Short-Term Memory (LSTM) network to encode the answer text. Then, we introduce an interaction enhancement layer consisting of reference-answer-to-student-answer and student-answer-toreference-answer attentions, and we combine the attentions and their inputs to form enhanced representations of answer texts. Finally, we introduce two Siamese Bi-LSTM networks to fuse the enhanced representations of answer texts and combine their multiple pooled vectors for grade classification on a multi-linear prediction layer. The experimental results show that our model significantly improves the performance of various simple models for Chinese and English ASAG tasks. The code is available online at https://github.com/wuhan-1222/DL_ASAG. INDEX TERMSAutomatic short-answer grading, textual entailment, hybrid neural networks, interaction enhancement.

show abstract

“…The challenge of automatically grading short answers was first posed a few decades ago. Earlier ASAG approaches consisted of clustering similar answers (Basu et al, 2013;Zehner et al, 2016), utilizing hand-crafted rules, schemes and ideal answer models (Leacock and Chodorow, 2003;Willis, 2015), or combining manually engineered features with various machine learning models (Marvaniya et al, 2018;Mohler et al, 2011;Saha et al, 2018;Sahu and Bhowmick, 2020;Sultan et al, 2016). Please refer to one of the comprehensive surveys of this field for a more in-depth elaboration of these approaches (Burrows et al, 2015;Galhardi and Brancher, 2018;Roy et al, 2015).…”

Section: Automatic Short Answer Gradingmentioning

confidence: 99%

Cheating Automatic Short Answer Grading: On the Adversarial Usage of Adjectives and Adverbs

Filighera¹,

Ochs²,

Steuer³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatic grading models are valued for the time and effort saved during the instruction of large student bodies. Especially with the increasing digitization of education and interest in large-scale standardized testing, the popularity of automatic grading has risen to the point where commercial solutions are widely available and used. However, for short answer formats, automatic grading is challenging due to natural language ambiguity and versatility. While automatic short answer grading models are beginning to compare to human performance on some datasets, their robustness, especially to adversarially manipulated data, is questionable. Exploitable vulnerabilities in grading models can have far-reaching consequences ranging from cheating students receiving undeserved credit to undermining automatic grading altogether -even when most predictions are valid. In this paper, we devise a black-box adversarial attack tailored to the educational short answer grading scenario to investigate the grading models' robustness. In our attack, we insert adjectives and adverbs into natural places of incorrect student answers, fooling the model into predicting them as correct. We observed a loss of prediction accuracy between 10 and 22 percentage points using the state-of-the-art models BERT and T5. While our attack made answers appear less natural to humans

show abstract

Creating Scoring Rubric from Representative Student Answers for Improved Short Answer Grading

Cited by 23 publications

References 22 publications

Reducing Workload in Short Answer Grading Using Machine Learning

Reducing Workload in Short Answer Grading Using Machine Learning

A Cross-Lingual Hybrid Neural Network With Interaction Enhancement for Grading Short-Answer Texts

Cheating Automatic Short Answer Grading: On the Adversarial Usage of Adjectives and Adverbs

Contact Info

Product

Resources

About