Automatic Assessment of the Speech of Young English Learners

Abstract

This paper introduces some of the research behind automatic scoring of the speaking part of the Arizona English Language Learner Assessment, a large-scale test now operational for students in Arizona. Approximately 70% of the students tested are in the range 4-11 years old. We cover the methods used to assess spoken responses automatically, considering both what the student says and the way in which the student speaks. We also provide evidence for the validity of machine scores. The assessments include 10 open-ended item types. For 9 of the 10 open item types, machine scoring performed at a similar level or better than human scoring at the item-type level. At the participant level, correlation coefficients between machine overall scores and average human overall scores were: Kindergarten: 0.88; Grades 1-2: 0.90; Grades 3-5: 0.94; Grades 6-8: 0.95; Grades 9-12: 0.93. The average correlation coefficient was 0.92. We include a note on implementing a detector to catch problematic test performances.

Deep neural network acoustic models for spoken assessment applications

“…The task types involved in this study are listed in Table 1. More details can be found in (Bernstein et al, 2010b;Cheng et al, 2014Cheng and Shen, 2010;Xu et al, 2012). Item types SentReading, SentRepeats, SentBuild, and PassReading are constrained: there is a set of pre-defined correct words and word sequences that test takers are expected to include in their response, although language models accept miscues and popular mistakes.…”

“…In the last decade, the tasks used in the automatic spoken assessment research have extended from constrained ones (such as reading or repeating a sentence, or reading a passage) to open-ended ones (such as open questions or retellings of a story, passage, picture or presentation) (Bernstein et al, 2010b;Cheng et al, 2014;Evanini and Wang, 2013;Nair et al, 2005;Pearson, 2009;Zechner et al, 2009a). Providing a reasonable recognition performance in these open-ended tasks is a big challenge.…”

Performance of Automated Speech Scoring on Different Low‐ to Medium‐Entropy Item Types for Low‐Proficiency English Learners

“…Cheng, D'Antilio, Chen, and Bernstein () presented an automated speech scoring system for an Arizona K–12 language test for English‐language learners that contained a range of items, from predictable (read or repeated prompt) to more open ended. There were 11 different item types, of which 9 were open ended.…”

Proceedings of the Workshop on Speech-Centric Natural Language Processing

“…Xie et al (2012) explored content measures based on the lexical similarity between the response and a set of reference responses. A content-scoring component based on word vectors was also part of the automated scoring engine described by Cheng et al (2014). In both these studies, content features were developed to supplement other features measuring various aspects of speaking proficiency.…”