Joint Learning of Domain Classification and Out-of-Domain Detection with Dynamic Class Weighting for Satisficing False Acceptance Rates

Kim, Joo-Kyung; Kim, Young–Bum

doi:10.21437/interspeech.2018-1581

Cited by 42 publications

(26 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another type of neural based OOD detection models aims to utilize a set of OOD data in the training phrase. Specifically, a special "OOD" label is added in a binary or multi-class classifier (e.g., [34]), and the inputs that fall into this special "OOD" class is rejected. However, the feasibility of this naive approach is limited in practice since suitable OOD data are usually hard to collect, and incorporating too many irrelevant OOD samples in training may cause a serious issue of data imbalance.…”

Section: Related Workmentioning

confidence: 99%

Out-of-Domain Detection for Natural Language Understanding in Dialog Systems

Zheng

Chen

Huang

2020

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Natural Language Understanding (NLU) is a vital component of dialogue systems, and its ability to detect Out-of-Domain (OOD) inputs is critical in practical applications, since the acceptance of the OOD input that is unsupported by the current system may lead to catastrophic failure. However, most existing OOD detection methods rely heavily on manually labeled OOD samples and cannot take full advantage of unlabeled data. This limits the feasibility of these models in practical applications.In this paper, we propose a novel model to generate highquality pseudo OOD samples that are akin to IN-Domain (IND) input utterances, and thereby improves the performance of OOD detection. To this end, an autoencoder is trained to map an input utterance into a latent code. and the codes of IND and OOD samples are trained to be indistinguishable by utilizing a generative adversarial network. To provide more supervision signals, an auxiliary classifier is introduced to regularize the generated OOD samples to have indistinguishable intent labels. Experiments show that these pseudo OOD samples generated by our model can be used to effectively improve OOD detection in NLU. Besides, we also demonstrate that the effectiveness of these pseudo OOD data can be further improved by efficiently utilizing unlabeled data.

show abstract

Section: Related Workmentioning

confidence: 99%

Out-of-Domain Detection for Natural Language Understanding in Dialog Systems

Zheng

Chen

Huang

2020

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…Text classification tasks in real-world applications often consists of 2 components-In-Doman (ID) classification and Out-of-Domain (OOD) detection components Kim and Kim, 2018;Shu et al, 2017;Shamekhi et al, 2018). ID classification refers to classifying a user's input with a label that exists in the training data, and OOD detection refers to designate a special OOD tag to the input when it does not belong to any of the labels in the ID training dataset (Dai et al, 2007).…”

Section: Introductionmentioning

confidence: 99%

Out-of-Domain Detection for Low-Resource Text Classification Tasks

Tan

Wang

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Out-of-domain (OOD) detection for lowresource text classification is a realistic but understudied task. The goal is to detect the OOD cases with limited in-domain (ID) training data, since we observe that training data is often insufficient in machine learning applications. In this work, we propose an OODresistant Prototypical Network to tackle this zero-shot OOD detection and few-shot ID classification task. Evaluation on real-world datasets show that the proposed solution outperforms state-of-the-art methods in zero-shot OOD detection task, while maintaining a competitive performance on ID classification task.

show abstract

“…For training classifiers, we initialize the embedding layer through publicly available GloVe [29] pre-trained word vectors (400 thousand words and 300 dimensions) 5 and further fine-tune the embedding layer through back-propagation. For BiLSTM model, we set the cell output dimension as 128 and dropout rate as 0.5.…”

Section: Hyperparametersmentioning

confidence: 99%

“…Only a few previous studies are related to unknown intent detection. For example, Kim and Kim [5] try to train an intent classifier and out-of-domain detector jointly, but they still need out-of-domain examples during the training process. Generative methods [6] try to generate positive and negative examples from known classes by using adversarial learning to augment training data.…”

Section: Introductionmentioning

confidence: 99%

A post-processing method for detecting unknown intent of dialogue system via pre-trained deep neural network classifier

Lin

2019

Knowledge-Based Systems

View full text Add to dashboard Cite

With the maturity and popularity of dialogue systems, detecting user's unknown intent in dialogue systems has become an important task. It is also one of the most challenging tasks since we can hardly get examples, prior knowledge or the exact numbers of unknown intents. In this paper, we propose SofterMax and deep novelty detection (SMDN), a simple yet effective post-processing method for detecting unknown intent in dialogue systems based on pre-trained deep neural network classifiers. Our method can be flexibly applied on top of any classifiers trained in deep neural networks without changing the model architecture. We calibrate the confidence of the softmax outputs to compute the calibrated confidence score (i.e., SofterMax) and use it to calculate the decision boundary for unknown intent detection. Furthermore, we feed the feature representations learned by the deep neural networks into traditional novelty detection algorithm to detect unknown intents from different perspectives. Finally, we combine the methods above to perform the joint prediction. Our method classifies examples that differ from known intents as unknown and does not require any examples or prior knowledge of it. We have conducted extensive experiments on three benchmark dialogue datasets. The results show that our method can yield significant improvements compared with the state-of-the-art baselines 1 .

show abstract

Joint Learning of Domain Classification and Out-of-Domain Detection with Dynamic Class Weighting for Satisficing False Acceptance Rates

Cited by 42 publications

References 26 publications

Out-of-Domain Detection for Natural Language Understanding in Dialog Systems

Out-of-Domain Detection for Natural Language Understanding in Dialog Systems

Out-of-Domain Detection for Low-Resource Text Classification Tasks

A post-processing method for detecting unknown intent of dialogue system via pre-trained deep neural network classifier

Contact Info

Product

Resources

About