Adversarial Self-Supervised Learning for Out-of-Domain Detection

Zeng, Zhiyuan; He, Keqing; Yan, Yuanmeng; Xu, Hong; Wang, Xu

doi:10.18653/v1/2021.naacl-main.447

Cited by 16 publications

(11 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In-domain Samples Contrastive Learning Recent contrastive learning methods have proven effective to learn unsupervised representations for downstream tasks. Winkens et al (2020); Zeng et al (2021b) combine cross-entropy loss on labeled IND data and instance-wise contrastive learning (CL) loss on unlabeled data (including unlabeled IND and OOD intents). They require a large amount of unlabeled corpus and can't explicitly distinguish different intent types.…”

Section: Encodermentioning

confidence: 99%

Revisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold

Wu¹,

He²,

Yan³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

Detecting Out-of-Domain (OOD) or unknown intents from user queries is essential in a taskoriented dialog system. A key challenge of OOD detection is the overconfidence of neural models. In this paper, we comprehensively analyze overconfidence and classify it into two perspectives: over-confident OOD and in-domain (IND). Then according to intrinsic reasons, we respectively propose a novel reassigned contrastive learning (RCL) to discriminate IND intents for over-confident OOD and an adaptive class-dependent local threshold mechanism to separate similar IND and OOD intents for overconfident IND. Experiments and analyses show the effectiveness of our proposed method for both aspects of overconfidence issues. 1

show abstract

Section: Encodermentioning

confidence: 99%

Revisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold

Wu¹,

He²,

Yan³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, Lin et al [6] and Yan et al [7] propose to first learn discriminative deep features through large margin cosine loss and Gaussian mixture loss, and then apply the local outlier factor algorithm to detect open intents. Xu et al [9] and Zeng et al [11] also propose to first learn discriminative deep features through large margin cosine loss and self-supervised contrastive loss, and then apply Mahalanobis distance to detect open intents. Considering the feature learning is optimized separately from decision boundary learning in these methods, Zhang et al [10] proposes a joint optimized adaptive decision boundary method for outlier detection.…”

Section: Open Intentsmentioning

confidence: 99%

“…Xu et al [9] proposes BiLSTM with LMLC as a feature extractor and uses Gaussian discriminant analysis (GDA) and Mahalanobis distance to detect open intents. Zeng et al [11] proposes a contrastive learning framework to model discriminative feature and also uses GDA and Mahalanobis distance to detect open intents. Similar to the post-processing methods, the joint-optimization methods also detect open intents in a post-processing way, but they consider to jointly learn the feature space and the decision boundary of outlier detection.…”

Section: A Open Intent Classificationmentioning

confidence: 99%

Learning to Classify Open Intent via Soft Labeling and Manifold Mixup

Cheng¹,

Jiang²,

Yin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Open intent classification is a practical yet challenging task in dialogue systems. Its objective is to accurately classify samples of known intents while at the same time detecting those of open (unknown) intents. Existing methods usually use outlier detection algorithms combined with K-class classifier to detect open intents, where K represents the class number of known intents. Different from them, in this paper, we consider another way without using outlier detection algorithms. Specifically, we directly train a (K+1)-class classifier for open intent classification, where the (K+1)-th class represents open intents. To address the challenge that training a (K+1)-class classifier with training samples of only K classes, we propose a deep model based on Soft Labeling and Manifold Mixup (SLMM). In our method, soft labeling is used to reshape the label distribution of the known intent samples, aiming at reducing model's overconfident on known intents. Manifold mixup is used to generate pseudo samples for open intents, aiming at well optimizing the decision boundary of open intents. Experiments on four benchmark datasets demonstrate that our method outperforms previous methods and achieves state-of-the-art performance. All the code and data of this work can be obtained at https://github.com/ zifengcheng/SLMM.

show abstract

“…The adversarial inputs lead to serious concerns about AI safety. However, despite the existing literature in adversarial training with SSL that works on algorithm design and empirical study, e.g., Kim et al (2020); Zeng et al (2021); Ho and Vasconcelos (2020); Gowal et al (2020); Chen et al (2020b), there is little theoretical understanding towards this.…”

Section: Introductionmentioning

confidence: 99%

Unlabeled Data Help: Minimax Analysis and Adversarial Robustness

Xing¹,

Song²,

Cheng³

2022

Preprint

View full text Add to dashboard Cite

The recent proposed self-supervised learning (SSL) approaches successfully demonstrate the great potential of supplementing learning algorithms with additional unlabeled data. However, it is still unclear whether the existing SSL algorithms can fully utilize the information of both labelled and unlabeled data. This paper gives an affirmative answer for the reconstruction-based SSL algorithm (Lee et al., 2020) under several statistical models. While existing literature only focuses on establishing the upper bound of the convergence rate, we provide a rigorous minimax analysis, and successfully justify the rate-optimality of the reconstruction-based SSL algorithm under different data generation models. Furthermore, we incorporate the reconstruction-based SSL into the existing adversarial training algorithms and show that learning from unlabeled data helps improve the robustness.Another motivation to study minimax lower bound is to understand the role of conditional independence (CI) for SSL. Lee et al. (2020) identifies that CI is a key factor yielding the estimation efficiency. A natural question would be how essential the CI condition is. Lee et al. (2020) suggests that the convergence rate of SSL estimate might be slower when CI does not hold. However, it is unclear whether it is caused by that the SSL is inefficient under conditional dependency, or that the fundamental information limit is worse under conditional dependency. To answer this question, minimax lower bound analysis is necessary.Besides the minimax analysis of SSL, since the aforementioned works observe a great advantage of SSL over classical supervised learning methods, it is natural

show abstract

Adversarial Self-Supervised Learning for Out-of-Domain Detection

Cited by 16 publications

References 24 publications

Revisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold

Revisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold

Learning to Classify Open Intent via Soft Labeling and Manifold Mixup

Unlabeled Data Help: Minimax Analysis and Adversarial Robustness

Contact Info

Product

Resources

About