URSABench: Comprehensive Benchmarking of Approximate Bayesian Inference Methods for Deep Neural Networks

Vadera, Meet P.; Cobb, Adam D.; Jalaian, Brian; Marlin, Benjamin M.

doi:10.48550/arxiv.2007.04466

Cited by 4 publications

(9 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, [12,11,13,14,15] presented large-scale evaluation studies of BDL methods with benchmarking on real-world datasets. These studies motivate data retention and distribution shift as generic protocols for evaluating predictive uncertainty.…”

Section: A Related Workmentioning

confidence: 99%

Benchmarking Scalable Predictive Uncertainty in Text Classification

Landeghem

Blaschko

Anckaert³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

show abstract

Section: A Related Workmentioning

confidence: 99%

Benchmarking Scalable Predictive Uncertainty in Text Classification

Landeghem

Blaschko

Anckaert³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…For our paper, we use the SGHMC implementation provided by Vadera et al [35]. We also utilize parts of the PyTorch implementation provided by Frankle and Carbin [10] for our experiments.…”

Section: A2 Implementation Detailsmentioning

confidence: 99%

Impact of Parameter Sparsity on Stochastic Gradient MCMC Methods for Bayesian Deep Learning

Vadera¹,

Cobb²,

Jalaian³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Bayesian methods hold significant promise for improving the uncertainty quantification ability and robustness of deep neural network models. Recent research has seen the investigation of a number of approximate Bayesian inference methods for deep neural networks, building on both the variational Bayesian and Markov chain Monte Carlo (MCMC) frameworks. A fundamental issue with MCMC methods is that the improvements they enable are obtained at the expense of increased computation time and model storage costs. In this paper, we investigate the potential of sparse network structures to flexibly trade-off model storage costs and inference run time against predictive performance and uncertainty quantification ability. We use stochastic gradient MCMC methods as the core Bayesian inference method and consider a variety of approaches for selecting sparse network structures. Surprisingly, our results show that certain classes of randomly selected substructures can perform as well as substructures derived from state-of-the-art iterative pruning methods while drastically reducing model training times.Preprint. Under review.

show abstract

“…As baselines we use Vanilla ResNet-20 and Deep Ensemble with ResNet-20 and M = 3 members. We used the Axolotl framework 2 and Keras for implementation.…”

Section: A Experimental Setup For Cifar-10mentioning

confidence: 99%

“…There exist many approaches towards uncertainty estimation, however, many of them are complex to train, lack good predictive performance, or are very resource-intensive [2]. One of the best performing approaches is Deep Ensemble [3], where multiple models are trained independently and used at inference, thus making it not viable on mobile platforms with low-latency requirements.…”

Section: Introductionmentioning

confidence: 99%

On Efficient Uncertainty Estimation for Resource-Constrained Mobile Applications

Rock¹,

Azevedo²,

Jong³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks have shown great success in prediction quality while reliable and robust uncertainty estimation remains a challenge. Predictive uncertainty supplements model predictions and enables improved functionality of downstream tasks including embedded and mobile applications, such as virtual reality, augmented reality, sensor fusion, and perception. These applications often require a compromise in complexity to obtain uncertainty estimates due to very limited memory and compute resources. We tackle this problem by building upon Monte Carlo Dropout (MCDO) models using the Axolotl framework; specifically, we diversify sampled subnetworks, leverage dropout patterns, and use a branching technique to improve predictive performance while maintaining fast computations. We conduct experiments on (1) a multi-class classification task using the CIFAR10 dataset, and (2) a more complex human body segmentation task. Our results show the effectiveness of our approach by reaching close to Deep Ensemble prediction quality and uncertainty estimation, while still achieving faster inference on resource-limited mobile platforms.

show abstract

URSABench: Comprehensive Benchmarking of Approximate Bayesian Inference Methods for Deep Neural Networks

Cited by 4 publications

References 12 publications

Benchmarking Scalable Predictive Uncertainty in Text Classification

Benchmarking Scalable Predictive Uncertainty in Text Classification

Impact of Parameter Sparsity on Stochastic Gradient MCMC Methods for Bayesian Deep Learning

On Efficient Uncertainty Estimation for Resource-Constrained Mobile Applications

Contact Info

Product

Resources

About