Off-Topic Spoken Response Detection Using Siamese Convolutional Neural Networks

Abstract

In this study, we developed an off-topic response detection system to be used in the context of the automated scoring of nonnative English speakers' spontaneous speech. Based on transcriptions generated from an ASR system trained on non-native speakers' speech and various semantic similarity features, the system classified each test response as an on-topic or off-topic response. The recent success of deep neural networks (DNN) in text similarity detection led us to explore DNN-based document similarity features. Specifically, we used a siamese adaptation of the convolutional network, due to its efficiency in learning similarity patterns simultaneously from both responses and questions used to elicit responses. In addition, a baseline system was developed using a standard vector space model (VSM) trained on sample responses for each question. The accuracy of the siamese CNN-based system was 0.97 and there was a 50% relative error reduction compared to the standard VSM-based system. Furthermore, the accuracy of the siamese CNN-based system was consistent across different questions.

Automatic Assessment of Speech Impairment in Cantonese-Speaking People with Aphasia

“…It is able to learn high-level representations of inputs from their respective sub-networks for further comparison. The inputs for Siamese network can be image data, sentences or sequential data, and thus the model is applicable to various comparison-making tasks such as image matching (e.g., face [40] and signature [41] verifications) and semantic matching (e.g., community question answering [42], off-topic response detection [43]). In this work, a siamese network is trained to compare spoken stories from impaired speakers and unimpaired ones for the text feature extraction.…”

“…The model architecture adopted in this study is motivated by that in [43], which is shown in Fig. 6.…”

Development of Robust Automated Scoring Models Using Adversarial Input for Oral Proficiency Assessment

“…In order to compare the performance of the content features with the speech-driven features, we trained three models: Speech (model based on 35 speech-driven features), Content (model based on 28 content features), and ALL (model based on both speech-driven and content features, 63 features in total). Finally, a total of 6 models (3 feature groups * 2 training data sets) were trained using the Random-ForestRegressor algorithm 6 in scikit-learn [20].…”

“…Even state-of-the art automated scoring models face challenges in scoring these atypical responses [1,2,3], and researchers in the automated scoring field have tried to solve this issue using a two-step approach where an automated filtering model, as a sub-module of an automated scoring system, filters out atypical responses, and only the remaining responses are scored by the scoring model. A spoken canned response detection system in [4], an off-topic response detection system in [5,6] and a coherence model in [7] are examples of this approach.…”

“…Similarly, synthesized atypical responses have been used to augment the training data of filtering models. [5,6] generated off-topic responses by mismatching questions and responses, and [7] generated incoherent responses by randomly shuffling the order of sentences in the original responses. They showed that the filtering models trained on the augmented data, including both normal responses and synthetic atypical responses, achieved promising performance in identifying synthetic atypical responses, but none of these studies reported the performance of the system on the authentic atypical responses.…”