Out-of-Domain Detection Method Based on Sentence Distance for Dialogue Systems

Oh, Kyo-Joong; Lee, Dongkun; Park, Chanyong; Jeong, Young-Seob; Hong, Sawook; Kwon, Sungtae; Choi, Ho-Jin

doi:10.1109/bigcomp.2018.00123

“…Shu et al (2017) create a binary classifier and calculate the confidence threshold for each class. Some distance-based methods (Oh et al, 2018;Lin and Xu, 2019;Yan et al, 2020) are also used to detect unknown intents as OOD utterances highly deviate from IND utterances in their local neighborhood. Simultaneously, with the advancement of deep generative models, learning such a model to approximate the distribution of training data is possible.…”

Section: Weighting Modulementioning

confidence: 99%

Energy-based Unknown Intent Detection with Data Manipulation

Ouyang¹,

Ye²,

Chen³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

8

1

0

View full text Add to dashboard Cite

Unknown intent detection aims to identify the out-of-distribution (OOD) utterance whose intent has never appeared in the training set. In this paper, we propose using energy scores for this task as the energy score is theoretically aligned with the density of the input and can be derived from any classifier. However, highquality OOD utterances are required during the training stage in order to shape the energy gap between OOD and in-distribution (IND), and these utterances are difficult to collect in practice. To tackle this problem, we propose a data manipulation framework to Generate high-quality OOD utterances with importance weighTs (GOT). Experimental results show that the energy-based detector fine-tuned by GOT can achieve state-of-the-art results on two benchmark datasets.

show abstract

“…Variants can tweak the embedding function or distance function used for determining the degree of separation. (Cavalin et al, 2020;Oh et al, 2018;Yilmaz and Toraman, 2020). For example, Local Outlier Factor (LOF) defines an outlier as a point whose density is lower than that of its nearest neighbors (Breunig et al, 2000;Lin and Xu, 2019).…”

Section: Probability Thresholdmentioning

confidence: 99%

GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentation

Chen¹,

Zhou²

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Practical dialogue systems require robust methods of detecting out-of-scope (OOS) utterances to avoid conversational breakdowns and related failure modes. Directly training a model with labeled OOS examples yields reasonable performance, but obtaining such data is a resource-intensive process. To tackle this limited-data problem, previous methods focus on better modeling the distribution of in-scope (INS) examples.We introduce GOLD as an orthogonal technique that augments existing data to train better OOS detectors operating in low-data regimes. GOLD generates pseudo-labeled candidates using samples from an auxiliary dataset and keeps only the most beneficial candidates for training through a novel filtering mechanism. In experiments across three target benchmarks, the top GOLD model outperforms all existing methods on all key metrics, achieving relative gains of 52.4%, 48.9% and 50.3% against median baseline performance.We also analyze the unique properties of OOS data to identify key factors for optimally applying our proposed method. 1

show abstract

“…Shu et al (2017) create a binary classifier and calculate the confidence threshold for each class. Some distance-based methods (Oh et al, 2018;Lin and Xu, 2019;Yan et al, 2020) are also used to detect unknown intents as OOD utterances highly deviate from IND utterances in their local neighborhood. Simultaneously, with the advancement of deep generative models, learning such a model to approximate the distribution of training data is possible.…”

Section: Weighting Modulementioning

confidence: 99%

Energy-based Unknown Intent Detection with Data Manipulation

Ouyang

¹

,

Ye

²

,

Chen

³

et al. 2021

Preprint

View full text Add to dashboard Cite

Unknown intent detection aims to identify the out-of-distribution (OOD) utterance whose intent has never appeared in the training set. In this paper, we propose using energy scores for this task as the energy score is theoretically aligned with the density of the input and can be derived from any classifier. However, highquality OOD utterances are required during the training stage in order to shape the energy gap between OOD and in-distribution (IND), and these utterances are difficult to collect in practice. To tackle this problem, we propose a data manipulation framework to Generate high-quality OOD utterances with importance weighTs (GOT). Experimental results show that the energy-based detector fine-tuned by GOT can achieve state-of-the-art results on two benchmark datasets.

show abstract

Out-of-Domain Detection Method Based on Sentence Distance for Dialogue Systems

Cited by 11 publications

References 3 publications

Energy-based Unknown Intent Detection with Data Manipulation

Energy-based Unknown Intent Detection with Data Manipulation

GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentation

Energy-based Unknown Intent Detection with Data Manipulation

Contact Info

Product

Resources

About