Automatic Detection of Off-Topic Spoken Responses Using Very Deep Convolutional Neural Networks

Wang, Xinhao; Yoon, Su‐Youn; Evanini, Keelan; Zechner, Klaus; Ye, Qian

doi:10.21437/interspeech.2019-1848

Cited by 5 publications

(17 citation statements)

References 16 publications

(19 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, the problem of robustness to prompts unseen in training is considered. The Hierarchical Attention-based Topic Model (HATM) (Malinin et al, 2017a) and Similarity Grid Model (Wang et al, 2019) are compared and extended. Universal regularisation and data augmentation, from structured techniques and machine translation, increased the HATM F 0.5 by 19% relative to 0.782 on the unseen evaluation set.…”

Section: Discussionmentioning

confidence: 99%

“…Automatic off-topic spoken response detection systems based on attention (Malinin et al, 2017b,a) and similarity grid (Wang et al, 2019) models have shown good performance for prompts seen in training. For operational reasons it would be cost and time effective to be able to use the same systems on responses to new prompts, unseen in training i.e.…”

Section: Introductionmentioning

confidence: 99%

“…has had some success with handling unseen prompts. This is still a challenging research problem, however, with significant degradation observed on even the best performing hierarchical attention-based topic model (HATM) (Malinin et al, 2017a), and with no assessment to date of Wang et al (2019)'s similarity grid model (SGM) approach. This paper therefore focuses on investigating how to improve performance on unseen prompts for these models.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Complementary Systems for Off-Topic Spoken Response Detection

Raina¹,

Gales²,

Knill³

2020

Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

Increased demand to learn English for business and education has led to growing interest in automatic spoken language assessment and teaching systems. With this shift to automated approaches it is important that systems reliably assess all aspects of a candidate's responses. This paper examines one form of spoken language assessment; whether the response from the candidate is relevant to the prompt provided. This will be referred to as off-topic spoken response detection. Two forms of previously proposed approaches are examined in this work: the hierarchical attention-based topic model (HATM); and the similarity grid model (SGM). The work focuses on the scenario when the prompt, and associated responses, have not been seen in the training data, enabling the system to be applied to new test scripts without the need to collect data or retrain the model. To improve the performance of the systems for unseen prompts, data augmentation based on easy data augmentation (EDA) and translation based approaches are applied. Additionally for the HATM, a form of prompt dropout is described. The systems were evaluated on both seen and unseen prompts from Linguaskill Business and General English tests. For unseen data the performance of the HATM was improved using data augmentation, in contrast to the SGM where no gains were obtained. The two approaches were found to be complementary to one another, yielding a combined F 0.5 score of 0.814 for off-topic response detection where the prompts have not been seen in training.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Complementary Systems for Off-Topic Spoken Response Detection

Raina¹,

Gales²,

Knill³

2020

Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

show abstract

“…Consequently, automated scoring systems tend to be more vulnerable than human raters to scoring inaccurately due to off-topic responses ( Lochbaum et al, 2013;Higgins and Heilman, 2014). To ensure the validity of speaking assessment scores, it is necessary to have a mechanism to flag off-topic responses before scores are reported (Wang et al, 2019). In our educational application, we use the automated speaking assessment system to help L2 learners prepare for the IELTS speaking test.…”

Section: Introductionmentioning

confidence: 99%

“…Lee et al (2017) proposed a siamese CNN to learn semantic differences between on-topic response-questions and off-topic response-questions. Wang et al (2019) proposed an approach based on similarity grids and deep CNN.…”

Section: Introductionmentioning

confidence: 99%

Gated Convolutional Bidirectional Attention-based Model for Off-topic Spoken Response Detection

Zha¹,

Li²,

Lin

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Off-topic spoken response detection, the task aiming at predicting whether a response is offtopic for the corresponding prompt, is important for an automated speaking assessment system. In many real-world educational applications, off-topic spoken response detectors are required to achieve high recall for off-topic responses not only on seen prompts but also on prompts that are unseen during training. In this paper, we propose a novel approach for offtopic spoken response detection with high offtopic recall on both seen and unseen prompts. We introduce a new model, Gated Convolutional Bidirectional Attention-based Model (GCBiA), which applies bi-attention mechanism and convolutions to extract topic words of prompts and key-phrases of responses, and introduces gated unit and residual connections between major layers to better represent the relevance of responses and prompts. Moreover, a new negative sampling method is proposed to augment training data. Experiment results demonstrate that our novel approach can achieve significant improvements in detecting off-topic responses with extremely high ontopic recall, for both seen and unseen prompts.

show abstract

Computerization of Off-Topic Essay Detection: A possibility?

Shahzad

Wali

2022

Educ Inf Technol

View full text Add to dashboard Cite

Checking essays written by students is a very time consuming task. Besides spellings and grammar, they also need to be evaluated on their semantic content such as cohesion, coherence, etc. In this study we focus on one such aspect of semantic content which is the topic of the essay. Putting it formally, given a prompt or essay-statement and an essay, this study aims to address the problem of predicting whether the essay is off-topic or not using machine learning techniques. With an increase in online learning and evaluation platforms especially during the COVID-19 pandemic, the off-topic detection system can be very useful to check essays that are mainly submitted online. In this paper, we answer the question: given a prompt and an essay written in Pakistani English, can the process of detecting whether the essay is off-topic or not be reliably and completely autonomized with zero human intervention using currently available tools and techniques? For this purpose, we explore and implement various embedding techniques proposed in recent years to extract similarity or dissimilarity features between question and response, and compare the performance of these techniques using 10 benchmark data sets and 6 classifiers. With different classifiers and different datasets along with different embeddings, we conclude that combining word mover distance, average word embeddings and idf weighted word embeddings together and then using random forest as the classifier is the best combination for off-topic essay detection. The accuracy obtained is 93.5%.

show abstract

Automatic Detection of Off-Topic Spoken Responses Using Very Deep Convolutional Neural Networks

Cited by 5 publications

References 16 publications

Complementary Systems for Off-Topic Spoken Response Detection

Complementary Systems for Off-Topic Spoken Response Detection

Gated Convolutional Bidirectional Attention-based Model for Off-topic Spoken Response Detection

Computerization of Off-Topic Essay Detection: A possibility?

Contact Info

Product

Resources

About