Off-topic Response Detection for Spontaneous Spoken English Assessment

Abstract

Automatic spoken language assessment systems are becoming increasingly important to meet the demand for English second language learning. This is a challenging task due to the high error rates of, even state-of-the-art, non-native speech recognition. Consequently current systems primarily assess fluency and pronunciation. However, content assessment is essential for full automation. As a first stage it is important to judge whether the speaker responds on topic to test questions designed to elicit spontaneous speech. Standard approaches to off-topic response detection assess similarity between the response and question based on bag-of-words representations. An alternative framework based on Recurrent Neural Network Language Models (RNNLM) is proposed in this paper. The RNNLM is adapted to the topic of each test question. It learns to associate example responses to questions with points in a topic space constructed using these example responses. Classification is done by ranking the topic-conditional posterior probabilities of a response. The RNNLMs associate a broad range of responses with each topic, incorporate sequence information and scale better with additional training data, unlike standard methods. On experiments conducted on data from the Business Language Testing Service (BULATS) this approach outperforms standard approaches.

Off-Topic Spoken Response Detection Using Siamese Convolutional Neural Networks

“…In this study, negative instances (authentic off-topic responses) were rare and difficult to collect. Similar to previous studies such as [10,3], we simulated negative instances by using responses elicited from different questions as off-topic responses. From a qualitative analysis of a small numbers of students' authentic off-topic responses, we found some evidence to support this approach for generating negative instances: some test takers did, in fact, appear to have used responses that had been prepared for other test questions.…”

“…However, in contrast to previous studies (e.g., [3]), we used a much larger number of questions in the set of off-topic responses. Although the number of questions we used is relatively large, it may still be substantially smaller than the number in an actual operational test, in which there may be no limitation on the number of topics for off-topic responses.…”

“…As a result, it may make it more difficult for the automated scoring system to provide a valid assessment of the proficiency of the speakers. In order to address this issue, previous studies such as [1,2,3] researched automated systems for filtering out off-topic responses. By filtering out these responses and preventing automated scoring systems from generating erroneous scores, the robustness of the automated scoring system can be improved.…”

“…For example, [18] examined various word embedding-based features in calculating similarities between questions and responses. It showed that naive cosine similarity measures using word embeddings could show lower performance than similarity measures using tf-idf s. [3] developed topic models based on Recurrent Neural Network language models (RNNLM). In this study, responses were categorized as off-topic when the confidence score of a response using a target topic model was not among the N-best solutions; the results showed that RNNLM models achieved a substantial improvement over the standard VSM-based approach.…”

“…Secondly, we use many more questions in evaluating off-topic detection models compared to previous studies. In [3], the assumption is made that students might use only a few limited topics for off-topic responses. Based on this assumption, closed-set off-topic detection was conducted using six question-based topic clustering models.…”