An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

Larson, Stefan; Mahendran, Anish; Peper, Joseph; Clarke, Christopher; Lee, Andrew; Hill, Parker; Kummerfeld, Jonathan K.; Leach, Kevin; Laurenzano, Michael A.; Tang, Lingjia; Mars, Jason

doi:10.48550/arxiv.1909.02027

Cited by 37 publications

(56 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DeepAligned [22] proposes a two-stage training strategy, in which the backbone is firstly trained to classify labeled samples for better representation (supervised learning) , and then secondly trained to classify samples with pseudo labels produced by K-Means iteratively. This paper has also conducted experiments on Clinc [12] and Banking [5], and randomly chooses a fraction of intents as known ones. DeepAligned [22] has outperformed other methods across all experiment settings and become the state-of-the-art model at the time.…”

Section: Related Workmentioning

confidence: 99%

“…Clinc. It contains 22500 samples of user queries in total and 150 unique labeled intents, which can be used for text classification or text clustering as well [12].…”

Section: Datasetsmentioning

confidence: 99%

“…In stage two, we build a classifier head for to produce pseudo labels, and train DCSC using Cross Entropy Loss and Supervised Contrastive Loss [10] on samples with either ground truth labels or pseudo labels. We conduct experiments on two public datasets, Clinc [12] and Banking [5]. To simulate the real situation and make our experiments comparable with previous works, we keep the same experiment settings as DeepAligned [22], and evaluate models using Accuracy (ACC), Adjusted Rand Index (ARI), Normalized Mutual Information (NMI).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Semi-Supervised Clustering with Contrastive Learning for Discovering New Intents

Feng¹,

Cheng²,

Hao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Most dialogue systems in real world rely on predefined intents and answers for QA service, so discovering potential intents from large corpus previously is really important for building such dialogue services. Considering that most scenarios have few intents known already and most intents waiting to be discovered, we focus on semi-supervised text clustering and try to make the proposed method benefit from labeled samples for better overall clustering performance. In this paper, we propose Deep Contrastive Semisupervised Clustering (DCSC), which aims to cluster text samples in a semi-supervised way and provide grouped intents to operation staff. To make DCSC fully utilize the limited known intents, we propose a two-stage training procedure for DCSC, in which DCSC will be trained on both labeled samples and unlabeled samples, and achieve better text representation and clustering performance. We conduct experiments on two public datasets to compare our model with several popular methods, and the results show DCSC achieve best performance across all datasets and circumstances, indicating the effect of the improvements in our work.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Clinc. It contains 22500 samples of user queries in total and 150 unique labeled intents, which can be used for text classification or text clustering as well [12].…”

Section: Datasetsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Semi-Supervised Clustering with Contrastive Learning for Discovering New Intents

Feng¹,

Cheng²,

Hao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…2. Training datasets include standard machine learning datasets -CIFAR (Krizhevsky et al, a,b), ImageNet (Russakovsky et al, 2015), and UCI (Dua and Graff, 2017) as well as more real-world problems -Clinc Intent Detection (Larson et al, 2019), Kaggle's Diabetic Retinopathy Detection (Filos et al, 2019), and Wikipedia Toxicity (Wulczyn et al, 2017). These span modalities such as tabular, text, and images.…”

Section: Uncertainty Baselinesmentioning

confidence: 99%

Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning

Nado¹,

Band²,

Collier³

et al. 2021

Preprint

View full text Add to dashboard Cite

High-quality estimates of uncertainty and robustness are crucial for numerous real-world applications, especially for deep learning which underlies many deployed ML systems. The ability to compare techniques for improving these estimates is therefore very important for research and practice alike. Yet, competitive comparisons of methods are often lacking due to a range of reasons, including: compute availability for extensive tuning, incorporation of sufficiently many baselines, and concrete documentation for reproducibility. In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-ofthe-art deep learning methods on a variety of tasks. As of this writing, the collection spans 19 methods across 9 tasks, each with at least 5 metrics. Each baseline is a self-contained experiment pipeline with easily reusable and extendable components. Our goal is to provide immediate starting points for experimentation with new methods or applications. Additionally we provide model checkpoints, experiment outputs as Python notebooks, and leaderboards for comparing results. https://github.com/google/uncertainty-baselines

show abstract

“…It is the inquire dataset of online banking, consists of 77 intents in a single domain. CLINC150 proposed by [6], contains 150 intents of 10 domains and 22500 total samples. Implementation Details.…”

Section: Experiments 41 Experimental Setupmentioning

confidence: 99%

Density-Based Dynamic Curriculum Learning for Intent Detection

Gong

Cao

Yuan

et al. 2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Pre-trained language models have achieved noticeable performance on the intent detection task. However, due to assigning an identical weight to each sample, they suffer from the overfitting of simple samples and the failure to learn complex samples well. To handle this problem, we propose a density-based dynamic curriculum learning model. Our model defines the sample's difficulty level according to their eigenvectors' density. In this way, we exploit the overall distribution of all samples' eigenvectors simultaneously. Then we apply a dynamic curriculum learning strategy, which pays distinct attention to samples of various difficulty levels and alters the proportion of samples during the training process. Through the above operation, simple samples are well-trained, and complex samples are enhanced. Experiments on three open datasets verify that the proposed density-based algorithm can distinguish simple and complex samples significantly. Besides, our model obtains obvious improvement over the strong baselines.

show abstract

An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

Cited by 37 publications

References 0 publications

Semi-Supervised Clustering with Contrastive Learning for Discovering New Intents

Semi-Supervised Clustering with Contrastive Learning for Discovering New Intents

Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning

Density-Based Dynamic Curriculum Learning for Intent Detection

Contact Info

Product

Resources

About