Out-of-Domain Detection for Low-Resource Text Classification Tasks

Abstract

Out-of-domain (OOD) detection for lowresource text classification is a realistic but understudied task. The goal is to detect the OOD cases with limited in-domain (ID) training data, since we observe that training data is often insufficient in machine learning applications. In this work, we propose an OODresistant Prototypical Network to tackle this zero-shot OOD detection and few-shot ID classification task. Evaluation on real-world datasets show that the proposed solution outperforms state-of-the-art methods in zero-shot OOD detection task, while maintaining a competitive performance on ID classification task.

Improving Out-of-Scope Detection in Intent Classification by Using Embeddings of the Word Graph Space of the Classes

“…We take into account a commonly-used metric for OOS dectection, i.e. equal error rate (EER) (Lane et al, 2007;Ryu et al, 2017Ryu et al, , 2018Tan et al, 2019), which corresponds to the classification error rate when the threshold θ is set to a value where false acceptance rate (FAR) and false rejection rate (FRR) are the closest. These two metrics are defined as: In addition, in-scope error rate (ISER) is considered to report IS performance, i.e.…”

“…These two metrics are defined as: In addition, in-scope error rate (ISER) is considered to report IS performance, i.e. the accuracy considering only IS samples, as the class error rate in (Tan et al, 2019). This metric is important to evaluate whether the alternative classification methods are able to keep up with the performance of their counterparts in the classification task.…”

“…Given that no specific domain information or structure are taken into account, those methods are roughly the same that have been previously applied for other classification problems (Fumera et al, 2003;Luckner and Homenda, 2014). Some recent effort has been put specifically for OOS sample detection for intent recognition, either by considering OOS data during the training process (Tan et al, 2019) or solely by improving in-scope sample representation by means of Auto Encoders (Ryu et al, 2017) and Generative Adversarial Neural Networks (GANs) (Ryu et al, 2018), which is more desirable since there is no reliance on tedious data gathering processes to represent unpredictable OOS inputs. The latter two methods are directly related to ours but, unfortunately, the lack of publicly-available source codes and datasets has made it a challenge to reproduce the methods for a fair direct comparison with ours.…”

“…There also exist approaches based on the assumption that OOS data can be collected and included in the training set (Tan et al, 2019;Larson et al, 2019). However, in practice, collecting OOS data can be a burden for intent classifier creation, which is generally carried out by domain experts and not by machine learning experts.…”

Boosting Low-Resource Intent Detection with in-Scope Prototypical Networks

“…Few previous studies are related to both zero-shot OOD detection and few-shot ID classification for low-resource intent detection, which has just begun. For instance, Prototypical Networks (ProtoNet) [10], a metricbased meta-learning for few-shot image classification, has been actively applied to text classification and OOD detection [11,12]. One major limitation of such a method for OOD detection is that it only considers the carefully selected datasets with metadata, like large-scale independent source domains [13].…”