An Effective Baseline for Robustness to Distributional Shift

Thulasidasan, Sunil; Thapa, Sushil; Dhaubhadel, Sayera; Chennupati, Gopinath; Bhattacharya, Tanmoy; Bilmes, Jeff

doi:10.1109/icmla52953.2021.00050

Cited by 17 publications

(17 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Following the benchmarks given in [16], we compare our method against three recent notable SOTA approaches: Ensemble of self supervised Leave-Out Classifiers (ELOC) [30], Generalized ODIN (GODIN) [12], and Deep Abstaining Classifier (DAC) [28]. It should be noted that both DAC and ELOC involve data treated as OoD during training.…”

Section: Benchmark Resultsmentioning

confidence: 99%

“…Out-of-distribution (OoD) detection is a binary classification of detecting inputs sampled from distribution different from training data [11]. Many existing methods rely on training or tuning with data labelled as OoD from other categories [28,32], adversaries [15,19] or the leave-out sub-set of training samples [30]. However, it is intractable to cover the full space of OoD particularly for data with large dimensions (e.g., an image) [20], resulting in that a supervised method capturing limited facets of OoD distribution hardly generalizes without a data selection bias [24].…”

Section: Introductionmentioning

confidence: 99%

“…However, it is intractable to cover the full space of OoD particularly for data with large dimensions (e.g., an image) [20], resulting in that a supervised method capturing limited facets of OoD distribution hardly generalizes without a data selection bias [24]. Meanwhile, the classifier's accuracy on in-distribution (ID) images would be fluctuated by introducing OoD data with additional training objectives [28,30]. Such factors expose the unsupervised nature of OoD detection.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detection

Zhou¹

2022

Preprint

View full text Add to dashboard Cite

In some scenarios, classifier requires detecting out-ofdistribution samples far from its training data. With desirable characteristics, reconstruction autoencoder-based methods deal with this problem by using input reconstruction error as a metric of novelty vs. normality. We formulate the essence of such approach as a quadruplet domain translation with an intrinsic bias to only query for a proxy of conditional data uncertainty. Accordingly, an improvement direction is formalized as maximumly compressing the autoencoder's latent space while ensuring its reconstructive power for acting as a described domain translator. From it, strategies are introduced including semantic reconstruction, data certainty decomposition and normalized L2 distance to substantially improve original methods, which together establish state-of-the-art performance on various benchmarks, e.g., the FPR@95%TPR of CIFAR-100 vs. TinyImagenet-crop on Wide-ResNet is 0.2%. Importantly, our method works without any additional data, hardto-implement structure, time-consuming pipeline, and even harming the classification accuracy of known classes.

show abstract

Section: Benchmark Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detection

Zhou¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Both these methods use only the inlier samples for training. From the OOD literature, we include both methods which train only on the inlier samples, namely, MaxLogit (Hendrickx et al, 2021), energy-based scorer (Hendrickx et al, 2021), ODIN (Hendrickx et al, 2021), and SIRC (Xia & Bouganis, 2022), and methods which additionally use unlabeled samples, such as the coupled CE loss (CCE) of Thulasidasan et al (2021), the de-coupled CE loss (DCE) of Bitterwolf et al (2022), and the outlier exposure (OE) loss of Hendrycks et al (2019). The closest among these to our approach is SIRC, which also seeks to treat inlier abstentions differently from OOD abstentions, but uses a heuristic post-hoc rule.…”

Section: Mixing Proportionsmentioning

confidence: 99%

Learning to reject meets OOD detection: Are all abstentions created equal?

Narasimhan¹,

Menon²,

Jitkrittum³

et al. 2023

Preprint

View full text Add to dashboard Cite

Learning to reject (L2R) and out-of-distribution (OOD) detection are two classical problems, each involving the detection of certain abnormal samples: the goal is to detect "hard" samples on which to abstain in L2R, and "outlier" samples not drawn from the training distribution in OOD detection. Intriguingly, despite being developed in parallel literatures, both problems share a simple baseline: the maximum softmax probability (MSP) score. Despite this connection, there is limited understanding of precisely how these problems relate. In this paper, we formally relate these problems, and show how they may be jointly solved. We first show that while MSP is theoretically optimal for L2R, it can be sub-optimal for OOD detection in some important settings. We then characterize the Bayes-optimal classifier for a unified formulation that generalizes both L2R and OOD detection. Based on this, we design a plug-in approach for learning to abstain on both inlier and OOD samples, while constraining the total abstention budget. Experiments on benchmark OOD datasets demonstrate that our approach yields competitive classification and OOD detection performance compared to baselines from both literatures.

show abstract

“…ImageNet-O [9] and providing analysis for the same. As there are a plethora of OOD approaches out there [ [5], [15], [8], [13], [12], [28], [20], [11], [7]] and many more. We have included the predictive score based approach of Maximum Softmax Probability (MSP) [5] that is widely used across OOD detection literature as the baseline.…”

Section: Introductionmentioning

confidence: 99%

Out of Distribution Detection on ImageNet-O

Srivastava¹,

Jain²,

Thigle³

2022

Preprint

View full text Add to dashboard Cite

Out of distribution (OOD) detection is a crucial part of making machine learning systems robust. The is an important tool in testing the robustness of ImageNet [4] trained deep neural networks that are widely used across a variety of systems and applications. Inspired by [27], we aim to perform a comparative analysis of OOD detection methods on ImageNet-O [9], a first of its kind dataset with a label distribution different than that of Im-ageNet, that has been created to aid research in OOD detection for ImageNet [4] models. As this dataset is fairly new, we aim to provide a comprehensive benchmarking of some of the current state of the art OOD detection methods on this novel dataset. This benchmarking covers a variety of model architectures, settings where we haves prior access to the OOD data versus when we don't, predictive score based approaches, deep generative approaches to OOD detection, and more. The code is available here.

show abstract

An Effective Baseline for Robustness to Distributional Shift

Cited by 17 publications

References 20 publications

Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detection

Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detection

Learning to reject meets OOD detection: Are all abstentions created equal?

Out of Distribution Detection on ImageNet-O

Contact Info

Product

Resources

About