Modeling Discriminative Representations for Out-of-Domain Detection with Supervised Contrastive Learning

Abstract

Detecting Out-of-Domain (OOD) or unknown intents from user queries is essential in a taskoriented dialog system. A key challenge of OOD detection is to learn discriminative semantic features. Traditional cross-entropy loss only focuses on whether a sample is correctly classified, and does not explicitly distinguish the margins between categories. In this paper, we propose a supervised contrastive learning objective to minimize intra-class variance by pulling together in-domain intents belonging to the same class and maximize inter-class variance by pushing apart samples from different classes. Besides, we employ an adversarial augmentation mechanism to obtain pseudo diverse views of a sample in the latent space. Experiments on two public datasets prove the effectiveness of our method capturing discriminative representations for OOD detection. 1

Disentangled Knowledge Transfer for OOD Intent Discovery with Unified Contrastive Learning

“…We show the detailed statistics of CLINC (Larson et al, 2019) and BANKING (Casanueva et al, 2020) In real scenarios, we can use OOD detection models Zeng et al, 2021) to collect high-quality OOD data for OOD intent discovery.…”

“…Specifically, we firstly learn intent features using a context encoder like BERT, then add two independent transformation heads (instance-level head f and class-level head g) on top of BERT. In the IND pre-training stage, we use the head f to perform supervised instance-level contrastive learning Khosla et al, 2020;Gunel et al, 2021;Zeng et al, 2021) and the head g to compute traditional classification loss like cross-entropy. In the OOD clustering stage, we employ similar objectives for these two heads where f is still used for instance-level contrastive learning and g is used to perform class(cluster)-level contrastive learning .…”

“…2 Then we use two independent transformation heads f and g to get two disentangled latent vectors f i = f (z i ) and g i = g(z i ). 3 On top of the instance-level head f , we perform supervised contrastive learning (SCL) (Khosla et al, 2020;Zeng et al, 2021) as follows:…”

Revisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold

“…Lin and Xu (2019) employs an unsupervised density-based novelty detection algorithm, local outlier factor (LOF) to detect OOD intents. Further, Zeng et al (2021a) proposes a supervised contrastive learning objective to learn discriminative intent features. However, these methods ignore the key challenge of OOD detection, over-confidence.…”

“…Out-of-domain (OOD) detection is a key component of the task-oriented dialogue system (Gnewuch et al, 2017;Akasaki and Kaji, 2017;Shum et al, 2018;Tulshan and Dhage, 2019). It aims to decide whether a user query falls outside the range of predefined supported intents and avoid performing wrong operations (Lin and Xu, 2019;Xu et al, 2020;Zeng et al, 2021a). Due to the complexity of annotating OOD intents, most work focus on unsupervised OOD detection where there is no labeled OOD data but only labeled in-domain (IND) data (Xu et al, 2020).…”