On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty

Amersfoort, Joost van; Smith, Liz; Jesson, Andrew; Key, Oscar; Gal, Yarin

doi:10.48550/arxiv.2102.11409

Cited by 25 publications

(71 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A major problem for this task and OOD detection in general is that outliers and inliers may be indistinguishable in the feature space [17]. Feature collapse [16] can be alleviated by forcing the model to learn more informative features. This can be implemented by supplying auxiliary self-supervised loss [17,28] as well as by training on negative data which can be sourced from real datasets [14,29,15] or sampled from generative models [18,20]…”

Section: Open-set Recognitionmentioning

confidence: 99%

“…However, the assumption that MC dropout corresponds to Bayesian model sampling may not be satisfied in practice. Anomalies can also be detected by estimating the likelihood in feature space [12], however that approach is vulnerable to feature collapse [16].…”

Section: Dense Open-set Recognitionmentioning

confidence: 99%

“…In any case, these methods are not suitable for real-time applications. Recognition in latent space [12] may be sensitive to feature collapse [16,17] due to relying on a pretrained classifier. Reliance on auxiliary negative data may introduce several kinds of undesired bias such as overoptimistic performance on some test images.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dense anomaly detection by robust learning on synthetic negative data

Grcić¹,

Bevandić²,

Šegvić³

2021

Preprint

View full text Add to dashboard Cite

Standard machine learning is unable to accommodate inputs which do not belong to the training distribution. The resulting models often give rise to confident incorrect predictions which may lead to devastating consequences. This problem is especially demanding in the context of dense prediction since input images may be only partially anomalous. Previous work has addressed dense anomaly detection by discriminative training on mixed-content images. We extend this approach with synthetic negative patches which simultaneously achieve high inlier likelihood and uniform discriminative prediction. We generate synthetic negatives with normalizing flows due to their outstanding distribution coverage and capability to generate samples at different resolutions. We also propose to detect anomalies according to a principled information-theoretic criterion which can be consistently applied through training and inference. The resulting models set the new state of the art on standard benchmarks and datasets in spite of minimal computational overhead and refraining from auxiliary negative data.

show abstract

Section: Open-set Recognitionmentioning

confidence: 99%

Section: Dense Open-set Recognitionmentioning

confidence: 99%

See 1 more Smart Citation

Dense anomaly detection by robust learning on synthetic negative data

Grcić¹,

Bevandić²,

Šegvić³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…During inference time, each sample would get logit predictions as well logit variances, both of which are utilized to compute predictive probabilities via mean-field approximation [Lu et al, 2020]. GNN-SNGP: incorporating distance-preserving feature extraction Due to feature collapse in feature extraction [Liu et al, 2020, van Amersfoort et al, 2021, neural representation may not faithfully preserve distance in the input manifold. Liu et al [2020] propose to preserve input distance in feature extraction by applying Spectral Normalization (SN) [Gouk et al, 2021, Miyato et al, 2018 to the residual networks.…”

Section: Gnn Baselinementioning

confidence: 99%

Reliable Graph Neural Networks for Drug Discovery Under Distributional Shift

Han

Lakshminarayanan

Liu

2021

Preprint

View full text Add to dashboard Cite

The concern of overconfident mispredictions under distributional shift demands extensive reliability research on Graph Neural Networks used in critical tasks in drug discovery. Here we first introduce CardioTox, a real-world benchmark on drug cardiotoxicity to facilitate such efforts. Our exploratory study shows overconfident mispredictions are often distant from training data. That leads us to develop distance-aware GNNs: GNN-SNGP. Through evaluation on CardioTox and three established benchmarks, we demonstrate GNN-SNGP's effectiveness in increasing distance-awareness, reducing overconfident mispredictions and making better calibrated predictions without sacrificing accuracy performance. Our ablation study further reveals the representation learned by GNN-SNGP improves distancepreservation over its base architecture and is one major factor for improvements.

show abstract

“…However, the problem of uncertainty estimation (UE) in the context of VMP has not been widely covered yet. Previous works [9], [10], [11] consider a limited number of methods and are either disconnected from state-of-the-art approaches in VMP or recent advances in UE field [12], [13], [14] or both. On the other hand, Bayesian Deep Learning field would benefit from benchmarking on large industrial datasets.…”

Section: Introductionmentioning

confidence: 99%

Estimating Uncertainty For Vehicle Motion Prediction on Yandex Shifts Dataset

Pustynnikov¹,

Eremeev²

2021

Preprint

View full text Add to dashboard Cite

Motion prediction of surrounding agents is an important task in context of autonomous driving since it is closely related to driver's safety. Vehicle Motion Prediction (VMP) track of Shifts Challenge ** focuses on developing models which are robust to distributional shift and able to measure uncertainty of their predictions. In this work we present the approach that significantly improved provided benchmark and took 2nd place on the leaderboard.

show abstract

On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty

Cited by 25 publications

References 8 publications

Dense anomaly detection by robust learning on synthetic negative data

Dense anomaly detection by robust learning on synthetic negative data

Reliable Graph Neural Networks for Drug Discovery Under Distributional Shift

Estimating Uncertainty For Vehicle Motion Prediction on Yandex Shifts Dataset

Contact Info

Product

Resources

About