An AI-Based System for Formative and Summative Assessment in Data Science Courses

Vittorini, Pierpaolo; Menini, Stefano; Tonelli, Sara

doi:10.1007/s40593-020-00230-2

Cited by 47 publications

(32 citation statements)

References 38 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our tuned model achieves an accuracy of about 86.5% when classifying all samples as either correct or incorrect. This is comparable to recent works with a much narrower scope both in terms of questions and languages [23,53]. -We propose deferring decisions for difficult answers to humans.…”

Section: Introductionsupporting

confidence: 83%

“…The student might not only fail the exam, but the entire course and in extreme cases might even have to repeat an entire year of school, meaning separation from existing classmates and psychological distress [13]. Current works, e.g., [23,53] (and also this study), show that about 85-90% of answers are graded correctly as correct/incorrect if all answers are autograded. Humans make errors as well, and the risk is higher for repetitive tasks like grading the same question dozens of times.…”

Section: Introductionmentioning

confidence: 63%

“…In contrast to our work, the dataset is only small, monolingual, and from a particular domain only. [53] also assesses the value of feedback provided by the autograder based on student responses to a questionnaire and learning outcomes. The paper reports a reduction in grading time when a teacher was able to use autograding results.…”

Section: Autogradingmentioning

confidence: 99%

“…Assessing multilingual models and large datasets from various subjects and a rich set of instructors is a new development. In contrast to [4,23,53] our design follows the philosophy that trustworthy results are more important than grading all questions. Furthermore, most works did not consider ethical concerns.…”

Section: Autogradingmentioning

confidence: 99%

See 3 more Smart Citations

Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

Schneider¹,

Richner²,

Riser³

2022

Preprint

View full text Add to dashboard Cite

Autograding short textual answers has become much more feasible due to the rise of NLP and the increased availability of question-answer pairs brought about by a shift to online education. Autograding performance is still inferior to human grading. The statistical and black-box nature of state-of-the-art machine learning models makes them untrustworthy, raising ethical concerns and limiting their practical utility. Furthermore, the evaluation of autograding is typically confined to small, monolingual datasets for a specific question type. This study uses a large dataset consisting of about 10 million question-answer pairs from multiple languages covering diverse fields such as math and language, and strong variation in question and answer syntax. We demonstrate the effectiveness of finetuning transformer models for autograding for such complex datasets. Our best hyperparameter-tuned model yields an accuracy of about 86.5%, comparable to the state-of-the-art models that are less general and more tuned to a specific type of question, subject, and language. More importantly, we address trust and ethical concerns. By involving humans in the autograding process, we show how to improve the accuracy of automatically graded answers, achieving accuracy equivalent to that of teaching assistants. We also show how teachers can effectively control the type of errors made by the system and how they can validate efficiently that the autograder's performance on individual exams is close to the expected performance.

show abstract

Section: Introductionsupporting

confidence: 83%

Section: Introductionmentioning

confidence: 63%

Section: Autogradingmentioning

confidence: 99%

Section: Autogradingmentioning

confidence: 99%

See 2 more Smart Citations

Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

Schneider¹,

Richner²,

Riser³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The student might not only fail the exam, but the entire course and in extreme cases might even have to repeat an entire year of school, meaning separation from existing classmates and psychological distress (Cornell et al 2006). Current works, e.g., Vittorini et al (2020); Hsu et al (2021) (and also this study), show that about 85-90% of answers are graded correctly as correct/incorrect if all answers are autograded. Humans make errors as well, and the risk is higher for repetitive tasks like grading the same question dozens of times.…”

Section: Introductionmentioning

confidence: 57%

Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

Schneider

Richner

Riser³

2022

Int J Artif Intell Educ

View full text Add to dashboard Cite

Autograding short textual answers has become much more feasible due to the rise of NLP and the increased availability of question-answer pairs brought about by a shift to online education. Autograding performance is still inferior to human grading. The statistical and black-box nature of state-of-the-art machine learning models makes them untrustworthy, raising ethical concerns and limiting their practical utility. Furthermore, the evaluation of autograding is typically confined to small, monolingual datasets for a specific question type. This study uses a large dataset consisting of about 10 million question-answer pairs from multiple languages covering diverse fields such as math and language, and strong variation in question and answer syntax. We demonstrate the effectiveness of fine-tuning transformer models for autograding for such complex datasets. Our best hyperparameter-tuned model yields an accuracy of about 86.5%, comparable to the state-of-the-art models that are less general and more tuned to a specific type of question, subject, and language. More importantly, we address trust and ethical concerns. By involving humans in the autograding process, we show how to improve the accuracy of automatically graded answers, achieving accuracy equivalent to that of teaching assistants. We also show how teachers can effectively control the type of errors made by the system and how they can validate efficiently that the autograder’s performance on individual exams is close to the expected performance.

show abstract