Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors

Dusenberry, Michael W.; Jerfel, Ghassen; Wen, Yeming; Ma, Yuanlin; Snoek, Jasper; Heller, Katherine; Lakshminarayanan, Balaji; Tran, Dustin

doi:10.48550/arxiv.2005.07186

Cited by 4 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although a few different priors have been proposed for BNNs, these were mostly designed for specific tasks (Atanov et al, 2018;Ghosh & Doshi-Velez, 2017;Overweg et al, 2019;Nalisnick, 2018;Cui et al, 2020;Hafner et al, 2020) or relied heavily on non-standard inference methods (Sun et al, 2019;Ma et al, 2019;Karaletsos & Bui, 2020;Pearce et al, 2020). Moreover, while many interesting distributions have been proposed as variational posteriors for BNNs (Louizos & Welling, 2017;Swiatkowski et al, 2020;Dusenberry et al, 2020;Aitchison et al, 2020), these approaches have still used Gaussian priors.…”

Section: Related Workmentioning

confidence: 99%

“…However, the most common choice of the prior for BNN weights is the simplest one: the isotropic Gaussian. Isotropic Gaussians are used across almost all fields of Bayesian deep learning, ranging from variational inference (Blundell et al, 2015;Dusenberry et al, 2020), to sampling-based inference (Zhang et al, 2019), and even to infinite networks (Lee et al, 2017;Garriga-Alonso et al, 2019). This is troubling, since isotropic Gaussian priors are almost certainly not the best choice.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bayesian Neural Network Priors Revisited

Fortuin¹,

Garriga-Alonso²,

Ober³

et al. 2021

Preprint

View full text Add to dashboard Cite

Isotropic Gaussian priors are the de facto standard for modern Bayesian neural network inference. However, such simplistic priors are unlikely to either accurately reflect our true beliefs about the weight distributions, or to give optimal performance. We study summary statistics of neural network weights in different networks trained using SGD. We find that fully connected networks (FCNNs) display heavytailed weight distributions, while convolutional neural network (CNN) weights display strong spatial correlations. Building these observations into the respective priors leads to improved performance on a variety of image classification datasets. Moreover, we find that these priors also mitigate the cold posterior effect in FCNNs, while in CNNs we see strong improvements at all temperatures, and hence no reduction in the cold posterior effect.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Bayesian Neural Network Priors Revisited

Fortuin¹,

Garriga-Alonso²,

Ober³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The strengths of these two approaches need to be combined to remedy their common problems. Dusenberry et al [273] devised a rank-1 parameterization of BNNs and also utilized mixture approximate posteriors to capture multiple modes. Rank-1 BNNs demonstrated stateof-the-art performance across out of distribution variants, calibration on the test sets, accuracy and log-likelihood.…”

Section: Other Uq Techniquesmentioning

confidence: 99%

A review of uncertainty quantification in deep learning: Techniques, applications and challenges

et al. 2021

View full text Add to dashboard Cite

“…Variational methods for BNNs (Peterson, 1987;Hinton and Van Camp, 1993;Blundell et al, 2015) differ in their choices of prior and belief distributions and inference algorithm. This includes hierarchical priors (Louizos and Welling, 2016;Ghosh and Doshi-Velez, 2017), data priors (Louizos and Welling, 2016;Hafner et al, 2019b;Sun et al, 2019), flexible posteriors (Louizos and Welling, 2016;Sun et al, 2017;Louizos and Welling, 2017;Zhang et al, 2018;Chang et al, 2019), low rank posteriors (Izmailov et al, 2018;Dusenberry et al, 2020), and improved inference algorithms (Wen et al, 2018;Immer et al, 2020). BNNs have been leveraged for RL for robustness (Okada et al, 2020;Tran et al, 2019) and exploration (Houthooft et al, 2016;Azizzadenesheli et al, 2018).…”

Section: Variational Inferencementioning

confidence: 99%

Action and Perception as Divergence Minimization

Hafner¹,

Ortéga²,

Ba³

et al. 2020

Preprint

View full text Add to dashboard Cite

We introduce a unified objective for action and perception of intelligent agents. Extending representation learning and control, we minimize the joint divergence between the combined system of agent and environment and a target distribution. Intuitively, such agents use perception to align their beliefs with the world, and use actions to align the world with their beliefs. Minimizing the joint divergence to an expressive target maximizes the mutual information between the agent's representations and inputs, thus inferring representations that are informative of past inputs and exploring future inputs that are informative of the representations. This lets us explain intrinsic objectives, such as representation learning, information gain, empowerment, and skill discovery from minimal assumptions. Moreover, interpreting the target distribution as a latent variable model suggests powerful world models as a path toward highly adaptive agents that seek large niches in their environments, rendering task rewards optional. The framework provides a common

show abstract

Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors

Cited by 4 publications

References 0 publications

Bayesian Neural Network Priors Revisited

Bayesian Neural Network Priors Revisited

A review of uncertainty quantification in deep learning: Techniques, applications and challenges

Action and Perception as Divergence Minimization

Contact Info

Product

Resources

About