Using NLP to Support Scalable Assessment of Short Free Text Responses

Willis, Alistair

doi:10.3115/v1/w15-0628

Cited by 14 publications

(5 citation statements)

References 15 publications

(11 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Smooth Inverse Frequency algorithm [2] trains models based on preexisting word embeddings like GloVe [45], weighting each word in the sentence according to inverse frequency. A number of papers also discuss the possibility of automating short answer grading using techniques such as graph alignment techniques [40], grading constraints [51], and inductive logic programming [53].…”

Section: Semantic Similarity Algorithmsmentioning

confidence: 99%

QuizBot

Ruan

Jiang

et al. 2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Advances in conversational AI have the potential to enable more engaging and effective ways to teach factual knowledge. To investigate this hypothesis, we created QuizBot, a dialogue-based agent that helps students learn factual knowledge in science, safety, and English vocabulary. We evaluated QuizBot with 76 students through two within-subject studies against a flashcard app, the traditional medium for learning factual knowledge. Though both systems used the same algorithm for sequencing materials, QuizBot led to students recognizing (and recalling) over 20% more correct answers than when students used the flashcard app. Using a conversational agent is more time consuming to practice with; but in a second study, of their own volition, students spent 2.6x more time learning with QuizBot than with flashcards and reported preferring it strongly for casual learning. Our results in this second study showed QuizBot yielded improved learning gains over flashcards on recall. These results suggest that educational chatbot systems may have beneficial use, particularly for learning outside of traditional settings. CCS CONCEPTS • Human-centered computing → Natural language interfaces; • Applied computing → Education; • Computing methodologies → Artificial intelligence.

show abstract

Section: Semantic Similarity Algorithmsmentioning

confidence: 99%

QuizBot

Ruan

Jiang

et al. 2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…responses that are indicative of a deep approach to learning (Biggs and Tang, 2011)). Recent examples include studies that use statistical textmining techniques such as topic modelling (Basu et al, 2013) or k-means clustering (Zesch et al, 2015) or rule-based techniques such as inferencing clustering rules from hand-coded sets of student responses (Willis, 2015). For example, Liu et al (2017) present a machine learning approach to provide automated feedback of English essays regarding aspects of writing such as grammar, spelling, sentence diversity, and structure.…”

Section: Developments In Automatic Assessmentmentioning

confidence: 99%

Assessment analysis: methods and implementation options for multiple-choice exams

Rintala

2023

IJIIE

View full text Add to dashboard Cite

Automatic assessment can reduce teacher workload and offer flexibility for students, but if the teacher does not assess the exams manually, the teacher's view of the students' competence and exam-related behaviours will be meagre. This drawback can be mitigated through appropriate analytics. To support the design of an analytics module for an electronic assessment system, this paper investigates what kinds of analyses are useful for multiplechoice exams and how the analysis can be implemented. Three types of analysis were found useful: 1) descriptive statistics of exam answers, 2) analysis of errors in answers and 3) analysis of students' exam-taking behaviours. Though these analyses are to some extent generalisable, analysis needs vary, for example, by time, exam type and user. Therefore, it is suggested that to enable user-specific analyses in a resource-efficient manner, assessment software providers should facilitate access to assessment data in a structured format.

show abstract

“…The challenge of automatically grading short answers was first posed a few decades ago. Earlier ASAG approaches consisted of clustering similar answers (Basu et al, 2013;Zehner et al, 2016), utilizing hand-crafted rules, schemes and ideal answer models (Leacock and Chodorow, 2003;Willis, 2015), or combining manually engineered features with various machine learning models (Marvaniya et al, 2018;Mohler et al, 2011;Saha et al, 2018;Sahu and Bhowmick, 2020;Sultan et al, 2016). Please refer to one of the comprehensive surveys of this field for a more in-depth elaboration of these approaches (Burrows et al, 2015;Galhardi and Brancher, 2018;Roy et al, 2015).…”

Section: Automatic Short Answer Gradingmentioning

confidence: 99%

Cheating Automatic Short Answer Grading: On the Adversarial Usage of Adjectives and Adverbs

Filighera¹,

Ochs²,

Steuer³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatic grading models are valued for the time and effort saved during the instruction of large student bodies. Especially with the increasing digitization of education and interest in large-scale standardized testing, the popularity of automatic grading has risen to the point where commercial solutions are widely available and used. However, for short answer formats, automatic grading is challenging due to natural language ambiguity and versatility. While automatic short answer grading models are beginning to compare to human performance on some datasets, their robustness, especially to adversarially manipulated data, is questionable. Exploitable vulnerabilities in grading models can have far-reaching consequences ranging from cheating students receiving undeserved credit to undermining automatic grading altogether -even when most predictions are valid. In this paper, we devise a black-box adversarial attack tailored to the educational short answer grading scenario to investigate the grading models' robustness. In our attack, we insert adjectives and adverbs into natural places of incorrect student answers, fooling the model into predicting them as correct. We observed a loss of prediction accuracy between 10 and 22 percentage points using the state-of-the-art models BERT and T5. While our attack made answers appear less natural to humans

show abstract

Using NLP to Support Scalable Assessment of Short Free Text Responses

Cited by 14 publications

References 15 publications

QuizBot

QuizBot

Assessment analysis: methods and implementation options for multiple-choice exams

Cheating Automatic Short Answer Grading: On the Adversarial Usage of Adjectives and Adverbs

Contact Info

Product

Resources

About