Using exemplar responses for training and evaluating automated speech scoring systems

Abstract

Automated scoring engines are usually trained and evaluated against human scores and compared to the benchmark of human-human agreement. In this paper we compare the performance of an automated speech scoring engine using two corpora: a corpus of almost 700,000 randomly sampled spoken responses with scores assigned by one or two raters during operational scoring, and a corpus of 16,500 exemplar responses with scores reviewed by multiple expert raters. We show that the choice of corpus used for model evaluation has a major effect on estimates of system performance with r varying between 0.64 and 0.80. Surprisingly, this is not the case for the choice of corpus for model training: when the training corpus is sufficiently large, the systems trained on different corpora showed almost identical performance when evaluated on the same corpus. We show that this effect is consistent across several learning algorithms. We conclude that evaluating the model on a corpus of exemplar responses if one is available provides additional evidence about system validity; at the same time, investing effort into creating a corpus of exemplar responses for model training is unlikely to lead to a substantial gain in model performance.

Equity Beyond Bias in Language Technologies for Education

“…Rubric-based writing has drawbacks like rigid formulation of tasks (Warner, 2018), and many applications of rubrics are rooted in a racialized history difficult for technology to escape (Dixon-Román et al, 2019). Bias creeps into rubric writing and scoring of training data, unless extensive countermeasures are taken to maintain reliability across student backgrounds and varied response types (Loukina et al, 2018;West-Smith et al, 2018). It also limits flexibility in task choice and response type from students, limiting students to writing styles that mirror the norms of the dominant school culture.…”

Empowering Teacher Learning with AI: Automated Evaluation of Teacher Attention to Student Ideas during Argumentation-focused Discussion

“…Second, while there is a strong alignment between total and indicator human scores (𝑟 > 0.8), there is only moderate inter-rater agreement on the total and indicator scores (𝑟 = 0.4-0.5). We plan to both improve the rubric scoring and to collect more human ratings per transcript to allow distillation of the average human judgment as well as identification of cases that are more or less controversial for human raters; [25] we argue that evaluation on uncontroversial (exemplar) cases provides important information for understanding system performance. Previous research provides evidence that machine learning models are sometimes able to ignore biases and idiosyncrasies of specific human raters and agree with humans better than humans agree with each other.…”

Text mining in education

“…Others Others This paper presents the NLI-PT, the first Portuguese dataset compiled for native language identification (NLI), the task of identifying an author's first language based on their second language writing. (Loukina, Zechner, Bruno, & Beigman Klebanov, 2018) Question Classification Evaluation This paper compares the performance of an automated speech scoring engine using two corpora. (Rudzewitz et al, 2018) Question Classification Feedback This paper presents a novel approach leveraging task information to generate the expected range of well-formed and ill-formed variability in learner answers along with the required diagnosis and feedback.…”