Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting

Cobb, Adam D.; Jalaian, Brian

doi:10.48550/arxiv.2010.06772

Cited by 12 publications

(17 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the classic work of Neal (1996), there have been a few recent attempts at using full-batch HMC in BNNs (e.g. ; Cobb & Jalaian, 2020;Wenzel et al, 2020). These studies tend to use relatively short trajectory lengths (generally not considering a number of leapfrog steps greater than 100), and tend to focus on relatively small datasets and networks.…”

Section: Related Workmentioning

confidence: 99%

What Are Bayesian Neural Network Posteriors Really Like?

Izmailov,

Vikram,

Hoffman

et al. 2021

Preprint

View full text Add to dashboard Cite

The posterior over Bayesian neural network (BNN) parameters is extremely high-dimensional and non-convex. For computational reasons, researchers approximate this posterior using inexpensive mini-batch methods such as meanfield variational inference or stochastic-gradient Markov chain Monte Carlo (SGMCMC). To investigate foundational questions in Bayesian deep learning, we instead use full-batch Hamiltonian Monte Carlo (HMC) on modern architectures. We show that (1) BNNs can achieve significant performance gains over standard training and deep ensembles; (2) a single long HMC chain can provide a comparable representation of the posterior to multiple shorter chains; (3) in contrast to recent studies, we find posterior tempering is not needed for near-optimal performance, with little evidence for a "cold posterior" effect, which we show is largely an artifact of data augmentation; (4) BMA performance is robust to the choice of prior scale, and relatively similar for diagonal Gaussian, mixture of Gaussian, and logistic priors; (5) Bayesian neural networks show surprisingly poor generalization under domain shift; (6) while cheaper alternatives such as deep ensembles and SGMCMC can provide good generalization, they provide distinct predictive distributions from HMC. Notably, deep ensemble predictive distributions are similarly close to HMC as standard SGLD, and closer than standard variational inference.

show abstract

Section: Related Workmentioning

confidence: 99%

What Are Bayesian Neural Network Posteriors Really Like?

Izmailov,

Vikram,

Hoffman

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Over the last decade, with improving computational resources and advanced MCMC strategies, advanced proposal distributions incorporating gradients have been applied, with Langevin and Hamiltonian MCMC [32,71]. However, only in last five years has there been progress in area of Bayesian neural networks with Hamiltonian MCMC [72], and graphic processing unit (GPU) implementation to enhance computation [73]. Recent work in area where Langevin MCMC methods have been used for neural networks include the use of parallel tempering MCMC for simple neural networks for pattern classifica-tion and time series prediction problems [36].…”

Section: Bayesian Deep Learningmentioning

confidence: 99%

Bayesian graph convolutional neural networks via tempered MCMC

Chandra,

Bhagat,

Maharana

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep learning models, such as convolutional neural networks, have long been applied to image and multi-media tasks, particularly those with structured data. More recently, there has been more attention to unstructured data that can be represented via graphs. These types of data are often found in health and medicine, social networks, and research data repositories. Graph convolutional neural networks have recently gained attention in the field of deep learning that takes advantage of graph-based data representation with automatic feature extraction via convolutions. Given popularity of these methods in a wide range of applications, robust uncertainty quantification is vital. This remains a challenge for large models and unstructured datasets. Bayesian inference provides a principled and robust approach to uncertainty quantification of model parameters for deep learning models. Although Bayesian inference has been used extensively elsewhere, its application to deep learning remains limited due to the computational requirements of the Markov Chain Monte Carlo (MCMC) methods. Recent advances in parallel computing and advanced proposal schemes in sampling, such as incorporating gradients has allowed Bayesian deep learning methods to be implemented. In this paper, we present Bayesian graph deep learning techniques that employ state-of-art methods such as tempered MCMC sampling and advanced proposal schemes. Our results show that Bayesian graph convolutional methods can provide accuracy similar to advanced learning methods while providing a better alternative for robust uncertainty quantification for key benchmark problems.

show abstract

“…The limitations of MCMC has been addressed with better computational resources and advanced proposal distributions, incorporating gradients [29,62]. Hamiltonian MCMC sampling methods have been used for Bayesian neural networks [63] with enhanced computation strategies [64]. Langevin MCMC methods have been used in implementation of Bayesian neural networks for pattern classification and time series prediction problems [65].…”

Section: Bayesian Deep Learningmentioning

confidence: 99%

Revisiting Bayesian Autoencoders with MCMC

Chandra,

Jain,

Maharana

et al. 2021

Preprint

View full text Add to dashboard Cite

Autoencoders gained popularity in the deep learning revolution given their ability to compress data and provide dimensionality reduction. Although prominent deep learning methods have been used to enhance autoencoders, the need to provide robust uncertainty quantification remains a challenge. This has been addressed with variational autoencoders so far. Bayesian inference via MCMC methods have faced limitations but recent advances with parallel computing and advanced proposal schemes that incorporate gradients have opened routes less travelled. In this paper, we present Bayesian autoencoders powered MCMC sampling implemented using parallel computing and Langevin gradient proposal scheme. Our proposed Bayesian autoencoder provides similar performance accuracy when compared to related methods from the literature, with the additional feature of robust uncertainty quantification in compressed datasets. This motivates further application of the Bayesian autoencoder framework for other deep learning models.

show abstract

Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting

Cited by 12 publications

References 20 publications

What Are Bayesian Neural Network Posteriors Really Like?

What Are Bayesian Neural Network Posteriors Really Like?

Bayesian graph convolutional neural networks via tempered MCMC

Revisiting Bayesian Autoencoders with MCMC

Contact Info

Product

Resources

About