Contrastive Training for Improved Out-of-Distribution Detection

Winkens, Jim; Bunel, Rudy; Roy, Abhijit Guha; Stanforth, Robert; Natarajan, Vivek; Ledsam, Joseph R.; MacWilliams, Patricia; Kohli, Pushmeet; Karthikesalingam, Alan; Kohl, Simon A. A.; Cemgil, Taylan; Eslami, S. M. Ali; Ronneberger, Olaf

doi:10.48550/arxiv.2007.05566

Cited by 50 publications

(100 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…5.2, we use the Expected Calibration Error (ECE) (Guo et al, 2017) with 15 bins and Negative Log Likelihood (NLL) which is a strictly proper scoring rule (Gneiting & Raftery, 2007). Datasets: CIFAR10 (Krizhevsky et al, a) and CI-FAR100 (Krizhevsky et al, b) are considered near OOD datasets (Winkens et al, 2020;Fort et al, 2021) to each other. SVHN (Netzer et al, 2011) is considered a far OOD dataset to both CIFAR10 and CIFAR100 due to its shift in both concept and style.…”

Section: Methodsmentioning

confidence: 99%

“…Generalized ODIN Hsu et al (2020) (also Techapanurak et al (2019)) includes an additional network in the last layer to improve OOD detection during training. There are many other interesting OOD detection approaches that have achieved state-of-the-art performance without OOD data such as using contrastive learning with various transformations (Winkens et al, 2020;Tack et al, 2020), training a deep ensemble of multiple models (Lakshminarayanan et al, 2016) and leveraging large pretrained models (Fort et al, 2021). They require extended training time, hyperparameter tuning and careful selections of transformations, whereas our method does not introduce any hyperparameters and has negligible influence on standard cross-entropy training time.…”

Section: Extended Related Workmentioning

confidence: 99%

“…They require extended training time, hyperparameter tuning and careful selections of transformations, whereas our method does not introduce any hyperparameters and has negligible influence on standard cross-entropy training time. For example, Winkens et al (2020) trains a constrastive model for 1200 epochs on CIFAR100 whereas our model requires only 200 epochs (same as standard classifier training) on the same dataset.…”

Section: Extended Related Workmentioning

confidence: 99%

“…This prevents us from more systematically studying of OOD detection. A recent attempt (Winkens et al, 2020;Fort et al, 2021) to…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data

Tian¹,

Hsu²,

Shen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Moving beyond testing on in-distribution data, works on Out-of-Distribution (OOD) detection have recently increased in popularity. A recent attempt to categorize OOD data introduces the concept of near and far OOD detection. Specifically, prior works define characteristics of OOD data in terms of detection difficulty. We propose to characterize the spectrum of OOD data using two types of distribution shifts: covariate shift and concept shift, where covariate shift corresponds to change in style, e.g., noise, and concept shift indicates change in semantics. This characterization reveals that sensitivity to each type of shift is important to the detection and confidence calibration of OOD data. Consequently, we investigate score functions that capture sensitivity to each type of dataset shift and methods that improve them. To this end, we theoretically derive two score functions for OOD detection, the covariate shift score and concept shift score, based on the decomposition of KL-divergence for both scores, and propose a geometricallyinspired method (Geometric ODIN) to improve OOD detection under both shifts with only in-distribution data. Additionally, the proposed method naturally leads to an expressive post-hoc calibration function which yields state-of-the-art calibration performance on both in-distribution and out-of-distribution data. We are the first to propose a method that works well across both OOD detection and calibration, and under different types of shifts. Specifically, we improve the previous state-of-the-art OOD detection by relatively 7% AUROC on CIFAR100 vs. SVHN and achieve the best calibration performance of 0.084 Expected Calibration Error on the corrupted CIFAR100C dataset. View project page at https://sites.google.com/view/geometric-decomposition.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Extended Related Workmentioning

confidence: 99%

Section: Extended Related Workmentioning

confidence: 99%

“…This prevents us from more systematically studying of OOD detection. A recent attempt (Winkens et al, 2020;Fort et al, 2021) to…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data

Tian¹,

Hsu²,

Shen³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Due to the recent use of self-supervised learning along with contrastive loss in several applications, attempts have been made to use contrastive learning to detect OoD samples. Winkens et al, [37] uses a famous method SIMCLR [10] for contrastive training. It treated different augmentations as positive and every other image as negative to learn semantically rich features.…”

Section: Related Workmentioning

confidence: 99%

Out of distribution detection for skin and malaria images

Zaida¹,

Ali²,

Ali³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks have shown promising results in disease detection and classification using medical image data. However, they still suffer from the challenges of handling real-world scenarios especially reliably detecting out-of-distribution (OoD) samples. We propose an approach to robustly classify OoD samples in skin and malaria images without the need to access labeled OoD samples during training. Specifically, we use metric learning along with logistic regression to force the deep networks to learn much rich class representative features. To guide the learning process against the OoD examples, we generate ID similar-looking examples by either removing class-specific salient regions in the image or permuting image parts and distancing them away from in-distribution samples. During inference time, the K-reciprocal nearest neighbor is employed to detect out-of-distribution samples. For skin cancer OoD detection, we employ two standard benchmark skin cancer ISIC datasets as ID, and six different datasets with varying difficulty levels were taken as out of distribution. For malaria OoD detection, we use the BBBC041 malaria dataset as ID and five different challenging datasets as out of distribution. We achieved state-of-the-art results, improving 5% and 4% in TNR@ TPR95% over the previous state-of-the-art for skin cancer and malaria OoD detection respectively.

show abstract

Anomaly detection in IR images of PV modules using supervised contrastive learning

Bommes

Hoffmann

Buerhop‐Lutz

et al. 2022

Progress in Photovoltaics

View full text Add to dashboard Cite

Increasing deployment of photovoltaic (PV) plants requires methods for automatic detection of faulty PV modules in modalities, such as infrared (IR) images. Recently, deep learning has become popular for this. However, related works typically sample train and test data from the same distribution ignoring the presence of domain shift between data of different PV plants. Instead, we frame fault detection as more realistic unsupervised domain adaptation problem where we train on labeled data of one source PV plant and make predictions on another target plant. We train a ResNet‐34 convolutional neural network with a supervised contrastive loss, on top of which we employ a k‐nearest neighbor classifier to detect anomalies. Our method achieves a satisfactory area under the receiver operating characteristic (AUROC) of 73.3% to 96.6% on nine combinations of four source and target datasets with 2.92 million IR images of which 8.5% are anomalous. It even outperforms a binary cross‐entropy classifier in some cases. With a fixed decision threshold, this results in 79.4% and 77.1% correctly classified normal and anomalous images, respectively. Most misclassified anomalies are of low severity, such as hot diodes and small hot spots. Our method is insensitive to hyperparameter settings, converges quickly, and reliably detects unknown types of anomalies making it well suited for practice. Possible uses are in automatic PV plant inspection systems or to streamline manual labeling of IR datasets by filtering out normal images. Furthermore, our work serves the community with a more realistic view on PV module fault detection using unsupervised domain adaptation to develop more performant methods with favorable generalization capabilities.

show abstract

Contrastive Training for Improved Out-of-Distribution Detection

Cited by 50 publications

References 15 publications

Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data

Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data

Out of distribution detection for skin and malaria images

Anomaly detection in IR images of PV modules using supervised contrastive learning

Contact Info

Product

Resources

About