How Good is the Bayes Posterior in Deep Neural Networks Really?

Wenzel, Florian; Roth, Kevin A.; Veeling, Bastiaan S.; Swiatkowski, Jakub; Tran, Linh M.; Mandt, Stephan; Snoek, Jasper; Salimans, Tim; Jenatton, Rodolphe; Nowozin, Sebastian

doi:10.48550/arxiv.2002.02405

Cited by 46 publications

(109 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is also corroborated in practice, where the BMA approximated via Eq. ( 8) outperforms point estimate procedures (see, e.g., [146] and references therein). Nevertheless, this result requires perfect model specification; i.e., that there exists a θ such that the predictive model p(u|x, D) induced by p(θ|D) matches exactly the data-generating process [145].…”

Section: A4 Posterior Tempering For Model Misspecificationmentioning

confidence: 99%

“…In this regard, it has been shown that in many cases the BMA performs worse than a point estimate; see, e.g., [146,147]. This is often attributed to model misspecification and is also studied in theoretical works [145].…”

Section: A4 Posterior Tempering For Model Misspecificationmentioning

confidence: 99%

“…In this regard, a technique that is often used in practice is posterior tempering, i.e., sampling θ values from p(θ|D) 1/τ instead of the true posterior (τ = 1), where τ is called temperature. Specifically, it has been reported in the literature that "cold" posteriors, τ < 1, perform better [146,148,149], although using a more informed prior can potentially remove this effect [143]. Cold posteriors can be interpreted as over-counting the available data using 1/τ replications of it, thus, making the posterior more concentrated.…”

Section: A4 Posterior Tempering For Model Misspecificationmentioning

confidence: 99%

“…Furthermore, for including posterior tempering, presented in Section A.4, the term inside the brackets in Eq. (B.6) can be divided by τ (the temperature), while the variance of η remains unchanged; see [146] for more information. The optimal value of τ can be obtained via cross-validation using one of the metrics of Section 5 and validation data.…”

Section: B21 Markov Chain Monte Carlo (Mcmc)mentioning

confidence: 99%

See 3 more Smart Citations

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed, which we will make available as open-source library of all codes included in this framework.

show abstract

Section: A4 Posterior Tempering For Model Misspecificationmentioning

confidence: 99%

Section: A4 Posterior Tempering For Model Misspecificationmentioning

confidence: 99%

Section: A4 Posterior Tempering For Model Misspecificationmentioning

confidence: 99%

Section: B21 Markov Chain Monte Carlo (Mcmc)mentioning

confidence: 99%

See 2 more Smart Citations

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…No batch normalization or data augmentation is used as they do not admit Bayesian interpretations (Wenzel et al, 2020). In all experiments the hyper-parameters were optimized only for the ground truth MCMC chain.…”

Section: Bayesian Neural Network Subspace Samplingmentioning

confidence: 99%

Online, Informative MCMC Thinning with Kernelized Stein Discrepancy

Hawkins¹,

Koppel²,

Zhang³

2022

Preprint

View full text Add to dashboard Cite

A fundamental challenge in Bayesian inference is efficient representation of a target distribution. Many non-parametric approaches do so by sampling a large number of points using variants of Markov Chain Monte Carlo (MCMC). We propose an MCMC variant that retains only those posterior samples which exceed a KSD threshold, which we call KSD Thinning. We establish the convergence and complexity tradeoffs for several settings of KSD Thinning as a function of the KSD threshold parameter, sample size, and other problem parameters. Finally, we provide experimental comparisons against other online nonparametric Bayesian methods that generate lowcomplexity posterior representations, and observe superior consistency/complexity tradeoffs. Code is available at github.com/colehawkins/ KSD-Thinning.

show abstract

Density regression and uncertainty quantification with Bayesian deep noise neural networks

Zhang

Liu

Kang

2023

Stat

View full text Add to dashboard Cite

Deep neural network (DNN) models have achieved state‐of‐the‐art predictive accuracy in a wide range of applications. However, it remains a challenging task to accurately quantify the uncertainty in DNN predictions, especially those of continuous outcomes. To this end, we propose the Bayesian deep noise neural network (B‐DeepNoise), which generalizes standard Bayesian DNNs by extending the random noise variable from the output layer to all hidden layers. Our model is capable of approximating highly complex predictive density functions and fully learn the possible random variation in the outcome variables. For posterior computation, we provide a closed‐form Gibbs sampling algorithm that circumvents tuning‐intensive Metropolis–Hastings methods. We establish a recursive representation of the predictive density and perform theoretical analysis on the predictive variance. Through extensive experiments, we demonstrate the superiority of B‐DeepNoise over existing methods in terms of density estimation and uncertainty quantification accuracy. A neuroimaging application is included to show our model's usefulness in scientific studies.

show abstract

How Good is the Bayes Posterior in Deep Neural Networks Really?

Cited by 46 publications

References 0 publications

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Online, Informative MCMC Thinning with Kernelized Stein Discrepancy

Density regression and uncertainty quantification with Bayesian deep noise neural networks

Contact Info

Product

Resources

About