Deep Classifiers with Label Noise Modeling and Distance Awareness

Abstract

Uncertainty estimation in deep learning has recently emerged as a crucial area of interest to advance reliability and robustness in safety-critical applications. While there have been many proposed methods that either focus on distance-aware model uncertainties for out-of-distribution detection or on input-dependent label uncertainties for in-distribution calibration, both of these types of uncertainty are often necessary. In this work, we propose the HetSNGP method for jointly modeling the model and data uncertainty. We show that our proposed model affords a favorable combination between these two complementary types of uncertainty and thus outperforms the baseline methods on some challenging out-of-distribution datasets, including CIFAR-100C, Imagenet-C, and Imagenet-A. Moreover, we propose HetSNGP Ensemble, an ensembled version of our method which provides an additional type of uncertainty and also outperforms other ensemble baselines.

Assaying Out-Of-Distribution Generalization in Transfer Learning

“…Orthogonally, several (less scalable) works advocated for leveraging the compositional (perhaps causal [44]) structure in the underlying data-generative process to introduce suitable inductive biases [45][46][47][48][49][50][51]. (4) Simultaneously, Bayesian approaches for uncertainty predictions have been proposed to improve model calibration [52][53][54][55][56][57][58][59][60] and robustness on new distributions [61][62][63]. Recent work, however, found that larger models were natively better calibrated [64].…”

Priors in Bayesian Deep Learning: A Review

“…The first attempt at this trained a deep belief network on the data and then used it as the kernel function (Salakhutdinov & Hinton, 2007), but later approaches optimised the neural network kernel directly using the marginal likelihood (Calandra et al, 2016), often in combination with sparse approximations (Wilson et al, 2016a) or stochastic variational inference (Wilson et al, 2016b) for scalability (see Equation 7). In this vein, it has recently been proposed to regularise the Lipschitzness of the used neural network, in order for the learned kernel to preserve distances between data points and thus improve its out-ofdistribution uncertainties (Liu et al, 2020;Fortuin et al, 2021a). While all these approaches still rely on the log determinant term in Equation 35to protect them from overfitting, it has been shown that this is unfortunately not effective enough when the employed neural networks are overparameterised (Ober et al, 2021).…”