Revisiting the Calibration of Modern Neural Networks

Minderer, Matthias; Djolonga, Josip; Romijnders, Rob; Hubis, Frances Ann; Zhai, Xiaohua; Houlsby, Neil; Tran, Dustin; Lučić, Mario

doi:10.48550/arxiv.2106.07998

Cited by 9 publications

(15 citation statements)

References 28 publications

(46 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, in detection and segmentation, the calibration of a model is mainly determined by its architecture, and not by its size. These observations are in line with the results in image classification [Minderer et al, 2021].…”

Section: Model Calibrationsupporting

confidence: 92%

“…Here, we extend the reliability study of CNNs and VTs [Minderer et al, 2021] for detection and segmentation and report the results for in-distribution data. Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) [Naeini et al, 2015] are common metrics used to measure the calibration error of a neural network in classification.…”

Section: Model Calibrationmentioning

confidence: 94%

“…[Naseer et al, 2021] further studied the texture-bias of VTs and CNNs for image classification. [Minderer et al, 2021] additionally investigated the model calibration of VTs and CNNs, and demonstrated that type of architecture is a major determinant of properties of calibration. However, there has been no study of the impact of VTs on generalizability, robustness, calibration, and texture-bias in dense prediction tasks, including detection and segmentation, when replacing CNNs as the feature extractor.…”

Section: Related Workmentioning

confidence: 99%

“…Reliable models are well-calibrated, which means their prediction confidences and the accuracy of those predictions are highly correlated. However, recent studies in classification have shown that highly accurate CNNs are poorly calibrated [Guo et al, 2017], and VTs are better calibrated than CNNs [Minderer et al, 2021].…”

Section: Model Calibrationmentioning

confidence: 99%

See 3 more Smart Citations

A Comprehensive Study of Vision Transformers on Dense Prediction Tasks

Kishaan¹,

Kathiresan²,

Arnav³

et al. 2022

Preprint

View full text Add to dashboard Cite

Convolutional Neural Networks (CNNs), architectures consisting of convolutional layers, have been the standard choice in vision tasks. Recent studies have shown that Vision Transformers (VTs), architectures based on self-attention modules, achieve comparable performance in challenging tasks such as object detection and semantic segmentation. However, the image processing mechanism of VTs is different from that of conventional CNNs. This poses several questions about their generalizability, robustness, reliability, and texture bias when used to extract features for complex tasks. To address these questions, we study and compare VT and CNN architectures as a feature extractor in object detection and semantic segmentation. Our extensive empirical results show that the features generated by VTs are more robust to distribution shifts, natural corruptions, and adversarial attacks in both tasks, whereas CNNs perform better at higher image resolutions in object detection. Furthermore, our results demonstrate that VTs in dense prediction tasks produce more reliable and less texture biased predictions.

show abstract

Section: Model Calibrationsupporting

confidence: 92%

Section: Model Calibrationmentioning

confidence: 94%

Section: Related Workmentioning

confidence: 99%

Section: Model Calibrationmentioning

confidence: 99%

See 2 more Smart Citations

A Comprehensive Study of Vision Transformers on Dense Prediction Tasks

Kishaan¹,

Kathiresan²,

Arnav³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…When a model is highly confident in its prediction yet it is not accurate, such classifier is overconfident; otherwise it is under-confident. It is well-known that the regular NNs are over-confident [23,36] and (non-DP) BNNs are more calibrated [35]. In Table 3 On MLP, the Gaussian prior (or weight decay) significantly improves the MCE, in the non-DP regime and furthermore in the DP regime.…”

Section: Methodsmentioning

confidence: 99%

Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability

Zhang¹,

Bu²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Bayesian neural network (BNN) allows for uncertainty quantification in prediction, offering an advantage over regular neural networks that has not been explored in the differential privacy (DP) framework. We fill this important gap by leveraging recent development in Bayesian deep learning and privacy accounting to offer a more precise analysis of the trade-off between privacy and accuracy in BNN. We propose three DP-BNNs that characterize the weight uncertainty for the same network architecture in distinct ways, namely DP-SGLD (via the noisy gradient method), DP-BBP (via changing the parameters of interest) and DP-MC Dropout (via the model architecture). Interestingly, we show a new equivalence between DP-SGD and DP-SGLD, implying that some non-Bayesian DP training naturally allows for uncertainty quantification. However, the hyperparameters such as learning rate and batch size, can have different or even opposite effects in DP-SGD and DP-SGLD.Extensive experiments are conducted to compare DP-BNNs, in terms of privacy guarantee, prediction accuracy, uncertainty quantification, calibration, computation speed, and generalizability to network architecture. As a result, we observe a new tradeoff between the privacy and the reliability. When compared to non-DP and non-Bayesian approaches, DP-SGLD is remarkably accurate under strong privacy guarantee, demonstrating the great potential of DP-BNN in real-world tasks.

show abstract

Invariant representation driven neural classifier for anti-QCD jet tagging

Cheng

Courville

2022

J. High Energ. Phys.

View full text Add to dashboard Cite

We leverage representation learning and the inductive bias in neural-net-based Standard Model jet classification tasks, to detect non-QCD signal jets. In establishing the framework for classification-based anomaly detection in jet physics, we demonstrate that, with a well-calibrated and powerful enough feature extractor, a well-trained mass-decorrelated supervised Standard Model neural jet classifier can serve as a strong generic anti-QCD jet tagger for effectively reducing the QCD background. Imposing data-augmented mass-invariance (and thus decoupling the dominant factor) not only facilitates background estimation, but also induces more substructure-aware representation learning. We are able to reach excellent tagging efficiencies for all the test signals considered. In the best case, we reach a background rejection rate of 51 and a significance improvement factor of 3.6 at 50% signal acceptance, with the jet mass decorrelated. This study indicates that supervised Standard Model jet classifiers have great potential in general new physics searches.

show abstract

Revisiting the Calibration of Modern Neural Networks

Cited by 9 publications

References 28 publications

A Comprehensive Study of Vision Transformers on Dense Prediction Tasks

A Comprehensive Study of Vision Transformers on Dense Prediction Tasks

Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability

Invariant representation driven neural classifier for anti-QCD jet tagging

Contact Info

Product

Resources

About