Revisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold

Wu, Yue; He, Ke; Yan, Yuanmeng; Gao, QiXiang; Zeng, Zhiyuan; Zheng, Fujia; Zhao, Lulu; Jiang, He; Wu, Wei; Wang, Xu

doi:10.18653/v1/2022.naacl-main.307

Cited by 5 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The predicted probabilities from the classifier are used as confidence estimates for OOD detection. • Correctness Ranking Loss (CRL) (Moon et al, 2020) is a regularization term added to the CE- (Wu et al, 2022a) discriminates over-confident OOD samples using adaptive class-dependent local threshold mechanism.…”

Section: Compared Methodsmentioning

confidence: 99%

SELFOOD: Self-Supervised Out-Of-Distribution Detection via Learning to Rank

Mekala,

Samavedhi,

Dong

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Deep neural classifiers trained with crossentropy (CE) loss often suffer from poor calibration, necessitating the task of out-ofdistribution (OOD) detection. Traditional supervised OOD detection methods require expensive manual annotation of in-distribution and OOD samples. To address the annotation bottleneck, we introduce SELFOOD, a selfsupervised OOD detection method that requires only in-distribution samples as supervision. We cast OOD detection as an inter-document intralabel (IDIL) ranking problem and train the classifier with our pairwise ranking loss, referred to as IDIL loss. Specifically, given a set of in-distribution documents and their labels, for each label, we train the classifier to rank the softmax scores of documents belonging to that label to be higher than the scores of documents that belong to other labels. Unlike CE loss, our IDIL loss reaches zero when the desired confidence ranking is achieved and gradients are backpropagated to decrease probabilities associated with incorrect labels rather than continuously increasing the probability of the correct label. Extensive experiments with several classifiers on multiple classification datasets demonstrate the effectiveness of our method in both coarse-and fine-grained settings.

show abstract

Section: Compared Methodsmentioning

confidence: 99%

SELFOOD: Self-Supervised Out-Of-Distribution Detection via Learning to Rank

Mekala,

Samavedhi,

Dong

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

show abstract

“…In the subsequent stages we determine which of the IS intent is the closest match. Binary Classification (IS/OOS): A binary classifier is trained using the IS examples and OOS examples as explored in Tax and Duin (1999) (Wu et al, 2022), (Shen et al, 2021), and density based approach (Lin and Xu, 2019).…”

Section: Multi-stage Oosmentioning

confidence: 99%

Distinguish Sense from Nonsense: Out-of-Scope Detection for Virtual Assistants

Qian¹,

Qi²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Out of Scope (OOS) detection in Conversational AI solutions enables a chatbot to handle a conversation gracefully when it is unable to make sense of the end-user query. Accurately tagging a query as out-of-domain is particularly hard in scenarios when the chatbot is not equipped to handle a topic which has semantic overlap with an existing topic it is trained on. We propose a simple yet effective OOS detection method that outperforms standard OOS detection methods in a real-world deployment of virtual assistants. We discuss the various design and deployment considerations for a cloud platform solution to train virtual assistants and deploy them at scale. Additionally, we propose a collection of datasets that replicates real-world scenarios and show comprehensive results in various settings using both offline and online evaluation metrics.

show abstract