'In-Between' Uncertainty in Bayesian Neural Networks

Foong, Andrew Y. K.; Li, Yingzhen; Hernández-Lobato, José Miguel; Turner, Richard E.

doi:10.48550/arxiv.1906.11537

Cited by 12 publications

(32 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…. The Bayesian predictive of the model with a better marginal likelihood has increasing uncertainty away from the data as often desired (Foong et al, 2019). The parameters found by our online algorithm give rise to a Bayesian predictive without further tuning of parameters after training which is required when using the Laplace approximation (Ritter et al, 2018;Kristiadi et al, 2020).…”

Section: C1 Illustrative Examplesmentioning

confidence: 95%

“…We compare our method to cross-validation on eight UCI (Dua & Graff, 2017) regression datasets following Hernández-Lobato & Adams (2015) and Foong et al (2019). In this setup, each dataset is split into 90% training and 10% testing data and a neural network with a single hidden layer, 50 neurons, and a ReLU activation is used.…”

Section: Uci Regressionmentioning

confidence: 99%

“…In this setup, each dataset is split into 90% training and 10% testing data and a neural network with a single hidden layer, 50 neurons, and a ReLU activation is used. We use the handtuned and cross-validated results reported by Foong et al (2019) who compare MAP, variational inference, and the Laplace approximation. They use 10% of the training data for validation and optimize hyperparameters by grid-search.…”

Section: Uci Regressionmentioning

confidence: 99%

“…The full GGN approximation performs the best, followed by the EF and Kronecker Laplace-GGN approximations. The cross-validation results are taken fromFoong et al (2019). Results within one standard error of the best performance are in bold.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning

Immer,

Bauer,

Fortuin

et al. 2021

Preprint

View full text Add to dashboard Cite

Marginal-likelihood based model-selection, even though promising, is rarely used in deep learning due to estimation difficulties. Instead, most approaches rely on validation data, which may not be readily available. In this work, we present a scalable marginal-likelihood estimation method to select both the hyperparameters and network architecture based on the training data alone. Some hyperparameters can be estimated online during training, simplifying the procedure. Our marginallikelihood estimate is based on Laplace's method and Gauss-Newton approximations to the Hessian, and it outperforms cross-validation and manualtuning on standard regression and image classification datasets, especially in terms of calibration and out-of-distribution detection. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable (e.g., in nonstationary settings).

show abstract

Section: C1 Illustrative Examplesmentioning

confidence: 95%

Section: Uci Regressionmentioning

confidence: 99%

Section: Uci Regressionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning

Immer,

Bauer,

Fortuin

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In this section we provide more details on each of the modelling steps along the example of a synthetic one-dimensional non-linear regression dataset. We use the setup from (Foong et al, 2019) with two clusters of inputs x 1 ∼ U[−1, −0.7], x 2 ∼ U[0.5, 1] and y ∼ N (cos(4x + 0.8), 0.1 2 ). The shaded area indicates up to three standard deviations from the predictive mean.…”

Section: Tyxe By Example: Non-linear Regressionmentioning

confidence: 99%

TyXe: Pyro-based Bayesian neural nets for Pytorch

Ritter¹,

Karaletsos²

2021

Preprint

View full text Add to dashboard Cite

We introduce TyXe, a Bayesian neural network library built on top of Pytorch and Pyro. Our leading design principle is to cleanly separate architecture, prior, inference and likelihood specification, allowing for a flexible workflow where users can quickly iterate over combinations of these components. In contrast to existing packages TyXe does not implement any layer classes, and instead relies on architectures defined in generic Pytorch code. TyXe then provides modular choices for canonical priors, variational guides, inference techniques, and layer selections for a Bayesian treatment of the specified architecture. Sampling tricks for variance reduction, such as local reparameterization or flipout, are implemented as effect handlers, which can be applied independently of other specifications. We showcase the ease of use of TyXe to explore Bayesian versions of popular models from various libraries: toy regression with a pure Pytorch neural network; large-scale image classification with torchvision ResNets; graph neural networks based on DGL; and Neural Radiance Fields built on top of Pytorch3D. Finally, we provide convenient abstractions for variational continual learning. In all cases the change from a deterministic to a Bayesian neural network comes with minimal modifications to existing code, offering a broad range of researchers and practitioners alike practical access to uncertainty estimation techniques. The library is available at https://github.com/TyXe-BDL/TyXe.

show abstract