Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks

Padhy, Shreyas; Nado, Zachary; Ren, Jie; Liu, Jeremiah; Snoek, Jasper; Lakshminarayanan, Balaji

doi:10.48550/arxiv.2007.05134

Cited by 14 publications

(30 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Classification methods For the classification objective function L cls , we use a softmax classifier with a cross-entropy loss (CE) or a One-Vs-All classifier with Distance Maximization loss (OVADM) [18]. We use the OVADM loss and the CE loss to analyze the effect of distance-based loss in OOD detection.…”

Section: Methodsmentioning

confidence: 99%

“…An out-of-distribution (OOD) detection task is to recognize outliers or anomalies, which do not follow the distribution of the training data. To address this problem, numerous methods have been proposed in classification tasks in several modalities, including computer vision [8,16,17,9,23], natural language processing [12,21], and two-dimensional real-number dataset (e.g., Gaussian noise distribution) [19,16,18,24]. Especially, [8,16,17] are simple yet efficient algorithms since they use existing trained models for OOD detection without additional fine-tuning with OOD samples.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation of Out-of-Distribution Detection Performance of Self-Supervised Learning in a Controllable Environment

Park¹,

Jo²,

Gwak³

et al. 2020

Preprint

View full text Add to dashboard Cite

We evaluate the out-of-distribution (OOD) detection performance of self-supervised learning (SSL) techniques with a new evaluation framework. Unlike the previous evaluation methods, the proposed framework adjusts the distance of OOD samples from the in-distribution samples. We evaluate an extensive combination of OOD detection algorithms on three different implementations of the proposed framework using simulated samples, images, and text. SSL methods consistently demonstrated the improved OOD detection performance in all evaluation settings.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Evaluation of Out-of-Distribution Detection Performance of Self-Supervised Learning in a Controllable Environment

Park¹,

Jo²,

Gwak³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The task assumes plenty of labeled inliers in training data. Padhy et al employ OVA-classifiers for this task [30]. Hendrycks et al reveal that exposing a model to outlier data allows it to effectively detect anomalies [18] and train it on out-of-distribution training data in a supervised way.…”

Section: Related Workmentioning

confidence: 99%

OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers

Saito,

Kim,

Saenko

2021

Preprint

View full text Add to dashboard Cite

Semi-supervised learning (SSL) is an effective means to leverage unlabeled data to improve a model's performance. Typical SSL methods like FixMatch assume that labeled and unlabeled data share the same label space. However, in practice, unlabeled data can contain categories unseen in the labeled set, i.e., outliers, which can significantly harm the performance of SSL algorithms. To address this problem, we propose a novel Open-set Semi-Supervised Learning (OSSL) approach called OpenMatch. Learning representations of inliers while rejecting outliers is essential for the success of OSSL. To this end, OpenMatch unifies FixMatch with novelty detection based on one-vs-all (OVA) classifiers. The OVA-classifier outputs the confidence score of a sample being an inlier, providing a threshold to detect outliers. Another key contribution is an open-set soft-consistency regularization loss, which enhances the smoothness of the OVA-classifier with respect to input transformations and greatly improves outlier detection. OpenMatch achieves stateof-the-art performance on three datasets, and even outperforms a fully supervised model in detecting outliers unseen in unlabeled data on CIFAR10.Preprint. Under review.

show abstract

“…To find similar metrics of OOD we explored general OOD literature for models which we can adapt for image segmentation. Variations of classical one-versus-all models have been adapted to neural networks Padhy et al (2020); Franchi et al (2020). The closest work that we could find to our proposed approach uses a deep network as a feature extractor for an RBF network van Amersfoort et al (2020).…”

Section: Related Workmentioning

confidence: 99%

“…In this section we evaluate our method alongside recent OOD models van Amersfoort et al (2020); Franchi et al (2020); Padhy et al (2020), assessing their capabilities to reach segmentation performance comparable to well-established deterministic models and whether they can accurately detect outliers.…”

Section: Evaluation On Brain Mrimentioning

confidence: 99%

Distributional Gaussian Process Layers for Outlier Detection in Image Segmentation

Popescu

Sharp

Cole

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a parameter efficient Bayesian layer for hierarchical convolutional Gaussian Processes that incorporates Gaussian Processes operating in Wasserstein-2 space to reliably propagate uncertainty. This directly replaces convolving Gaussian Processes with a distance-preserving affine operator on distributions. Our experiments on brain tissue-segmentation show that the resulting architecture approaches the performance of well-established deterministic segmentation algorithms (U-Net), which has never been achieved with previous hierarchical Gaussian Processes. Moreover, by applying the same segmentation model to out-of-distribution data (i.e., images with pathology such as brain tumors), we show that our uncertainty estimates result in out-of-distribution detection that outperforms the capabilities of previous Bayesian networks and reconstruction-based approaches that learn normative distributions.

show abstract

Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks

Cited by 14 publications

References 0 publications

Evaluation of Out-of-Distribution Detection Performance of Self-Supervised Learning in a Controllable Environment

Evaluation of Out-of-Distribution Detection Performance of Self-Supervised Learning in a Controllable Environment

OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers

Distributional Gaussian Process Layers for Outlier Detection in Image Segmentation

Contact Info

Product

Resources

About