Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users

Jospin, Laurent Valentin; Buntine, Wray; Boussaid, Farid; Laga, Hamid; Bennamoun, Mohammed

doi:10.48550/arxiv.2007.06823

Cited by 50 publications

(82 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many models have attempted to incorporate Bayesian elements into deep learning (see e.g. Bayesian networks [MacKay, 1992, Jospin et al, 2020, dropout recovering deep Gaussian process [Gal and Ghahramani, 2016], others [Osawa et al, 2019]). Often, these models require costly Monte Carlo training strategies, while the current approach can be trained with gradient descent.…”

Section: Introductionmentioning

confidence: 99%

Probabilistic partition of unity networks: clustering based deep approximation

Trask,

Gulian,

Huang

et al. 2021

Preprint

View full text Add to dashboard Cite

Partition of unity networks (POU-Nets) have been shown capable of realizing algebraic convergence rates for regression and solution of PDEs, but require empirical tuning of training parameters. We enrich POU-Nets with a Gaussian noise model to obtain a probabilistic generalization amenable to gradient-based minimization of a maximum likelihood loss. The resulting architecture provides spatial representations of both noiseless and noisy data as Gaussian mixtures with closed form expressions for variance which provides an estimator of local error. The training process yields remarkably sharp partitions of input space based upon correlation of function values. This classification of training points is amenable to a hierarchical refinement strategy that significantly improves the localization of the regression, allowing for higher-order polynomial approximation to be utilized. The framework scales more favorably to large data sets as compared to Gaussian process regression and allows for spatially varying uncertainty, leveraging the expressive power of deep neural networks while bypassing expensive training associated with other probabilistic deep learning methods. Compared to standard deep neural networks, the framework demonstrates hp-convergence without the use of regularizers to tune the localization of partitions. We provide benchmarks quantifying performance in high/low-dimensions, demonstrating that convergence rates depend only on the latent dimension of data within high-dimensional space. Finally, we introduce a new open-source data set of PDE-based simulations of a semiconductor device and perform unsupervised extraction of a physically interpretable reduced-order basis.

show abstract

Section: Introductionmentioning

confidence: 99%

Probabilistic partition of unity networks: clustering based deep approximation

Trask,

Gulian,

Huang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Alternative methods are based, indicatively, on ensembles of NN optimization iterates or independently trained NNs [39][40][41][42][43][44][45][46][47][48][49][50], as well as on the evidential framework [51][52][53][54][55][56][57][58][59]. Although Bayesian methods and ensembles are thoroughly discussed in this paper, the interested reader is also directed to the recent review studies in [60][61][62][63][64][65][66][67][68][69][70][71][72] for more information. Clearly, in the context of SciML, which may involve differential equations with unknown or uncertain terms and parameters, UQ becomes an even more demanding task; see Fig.…”

Section: Motivation and Scope Of The Papermentioning

confidence: 99%

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed, which we will make available as open-source library of all codes included in this framework.

show abstract

“…BNNs constitute a new direction in machine learning [16]. By connecting Bayesian statistics and deep learning, BNNs combine the benefits of Bayesian uncertainty quantification with the predictive power of NNs.…”

Section: A Model Selectionmentioning

confidence: 99%

Uncertainty Quantification in LV State Estimation Under High Shares of Flexible Resources

Müller¹,

Chevalier²,

Heinrich³

et al. 2021

Preprint

View full text Add to dashboard Cite

The ongoing electrification introduces new challenges to distribution system operators (DSOs). Controllable resources may simultaneously react to price signals, potentially leading to network violations. DSOs require reliable and accurate low-voltage state estimation (LVSE) to improve awareness and mitigate such events. However, the influence of flexibility activations on LVSE has not been addressed yet. It remains unclear if flexibility-induced uncertainty can be reliably quantified to enable robust DSO decision-making. In this work, uncertainty quantification in LVSE is systematically investigated for multiple scenarios of input availability and flexibility utilization, using real data. For that purpose, a Bayesian neural network (BNN) is compared to quantile regression. Results show that frequent flexibility activations can significantly deteriorate LVSE performance, unless secondary substation measurements are available. Moreover, it is demonstrated that the BNN captures flexibility-induced voltage drops by dynamically extending the prediction interval during activation periods, and that it improves interpretability regarding the cause of uncertainty.

show abstract

Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users

Cited by 50 publications

References 28 publications

Probabilistic partition of unity networks: clustering based deep approximation

Probabilistic partition of unity networks: clustering based deep approximation

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Uncertainty Quantification in LV State Estimation Under High Shares of Flexible Resources

Contact Info

Product

Resources

About