GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentation

Abstract

Practical dialogue systems require robust methods of detecting out-of-scope (OOS) utterances to avoid conversational breakdowns and related failure modes. Directly training a model with labeled OOS examples yields reasonable performance, but obtaining such data is a resource-intensive process. To tackle this limited-data problem, previous methods focus on better modeling the distribution of in-scope (INS) examples.We introduce GOLD as an orthogonal technique that augments existing data to train better OOS detectors operating in low-data regimes. GOLD generates pseudo-labeled candidates using samples from an auxiliary dataset and keeps only the most beneficial candidates for training through a novel filtering mechanism. In experiments across three target benchmarks, the top GOLD model outperforms all existing methods on all key metrics, achieving relative gains of 52.4%, 48.9% and 50.3% against median baseline performance.We also analyze the unique properties of OOS data to identify key factors for optimally applying our proposed method. 1

SILVER: Self Data Augmentation for Out-of-Scope Detection in Dialogues

“…As noted by Chen and Yu (2021), PersonaChat is particularly suitable for OOS detection of STAR data because it is a rich source of OOS dialogues. For FLOW, "Int."…”

“…with Data Augmentation GOLD (Chen and Yu, 2021) is the data augmentation method most closely related to this work. Given a small set of annotated OOS dialogues (1% of the size of INS), GOLD replaces utterances with sentences selected from an external pool to generate new OOS dialogues.…”

“…One such method is GOLD (Chen and Yu, 2021). GOLD uses simple rules to replace utterances in known OOS dialogues with sentences selected from a large pool, making it possible to train a binary classifier to decide OOS dialogues directly.…”

Metric Learning and Adaptive Boundary for Out-of-Domain Detection

“…The same paper shows that we can set OOD examples as n + 1 class and train the classification model with other IND classes. The aforementioned approaches can be used on artificially created OOD instances from IND training examples [17] or enlarge known OOD training data [3] with the help of a pretrained language model.…”

Data Augmentation for Intent Classification

“…Latent perturbation maps text to a hidden state before mapping back to natural language text again (Zhao et al, 2018). Auxiliary datasets take advantage of external unlabeled data from a relevant domain to form new pseudo-labeled examples (Chen and Yu, 2021). Text generation uses large pre-trained models to create new examples (Devlin et al, 2018).…”