The Connection between Bayesian Inference and Information Theory for Model Selection, Information Gain and Experimental Design

Oladyshkin, Sergey; Nowak, Wolfgang

doi:10.3390/e21111081

Cited by 21 publications

(61 citation statements)

References 54 publications

(133 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Estimating the relative entropy in Equation ( 13 ) usually requires a multidimensional integration that is often infeasible for most applied problems. However, employing Equation ( 13 ) and definition (5) from the recent findings in the paper [ 40 ], we avoid this multidimensional integration:

…”

Section: Bayesian Inference With Information Theory For a Gaussianmentioning

confidence: 99%

“…Therefore, employing Equation ( 15 ), information entropy can be directly estimated according to Equation ( A3 ) in the paper [ 40 ] using Monte Carlo sampling techniques on the GPE:

…”

Section: Bayesian Inference With Information Theory For a Gaussianmentioning

confidence: 99%

“…Therefore, the current Section 3 focuses on iterative selection of training points. It employs the link between Bayesian inference and information theory [ 40 ] similar to Section 2 in order to perform Bayesian active learning (BAL). The later will iteratively select new training point as the regions with required local accuracy becomes progressively clear during the Bayesian updating of a Gaussian process emulator described in Section 2 .…”

Section: Bayesian Active Learning For Gaussian Process Emulators Imentioning

confidence: 99%

“…A recent trend toward stochastic calibration based on surrogate models offers iterative improvement of surrogate representations [ 35 , 36 , 37 , 38 , 39 ] within a limited simulation time budget. For example, the link between Bayesian inference and information theory introduced in [ 40 ] can help localize adaptively the relevant spots in the parameter space for surrogate training, according to information-theoretic arguments. Such a procedure of active learning should be very informative regarding the available observation data.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Bayesian3 Active Learning for the Gaussian Process Emulator Using Information Theory

Oladyshkin

Mohammadi

Kroeker

et al. 2020

Entropy

Self Cite

View full text Add to dashboard Cite

Gaussian process emulators (GPE) are a machine learning approach that replicates computational demanding models using training runs of that model. Constructing such a surrogate is very challenging and, in the context of Bayesian inference, the training runs should be well invested. The current paper offers a fully Bayesian view on GPEs for Bayesian inference accompanied by Bayesian active learning (BAL). We introduce three BAL strategies that adaptively identify training sets for the GPE using information-theoretic arguments. The first strategy relies on Bayesian model evidence that indicates the GPE’s quality of matching the measurement data, the second strategy is based on relative entropy that indicates the relative information gain for the GPE, and the third is founded on information entropy that indicates the missing information in the GPE. We illustrate the performance of our three strategies using analytical- and carbon-dioxide benchmarks. The paper shows evidence of convergence against a reference solution and demonstrates quantification of post-calibration uncertainty by comparing the introduced three strategies. We conclude that Bayesian model evidence-based and relative entropy-based strategies outperform the entropy-based strategy because the latter can be misleading during the BAL. The relative entropy-based strategy demonstrates superior performance to the Bayesian model evidence-based strategy.

show abstract

…”

Section: Bayesian Inference With Information Theory For a Gaussianmentioning

confidence: 99%

“…Therefore, employing Equation ( 15 ), information entropy can be directly estimated according to Equation ( A3 ) in the paper [ 40 ] using Monte Carlo sampling techniques on the GPE:

…”

Section: Bayesian Inference With Information Theory For a Gaussianmentioning

confidence: 99%

Section: Bayesian Active Learning For Gaussian Process Emulators Imentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Bayesian3 Active Learning for the Gaussian Process Emulator Using Information Theory

Oladyshkin

Mohammadi

Kroeker

et al. 2020

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…Hence, analyzing the regional information entropy relationship can provide us with a decision basis for screening information. As a way of measuring the information content, information entropy S [ 34 ] is given by

where x is the output of the system and

is the probability distribution function (PDF) of x . There is a positive correlation between S and information content.…”

Section: The Regional Surrogate Model Technique Based On the Regiomentioning

confidence: 99%

Rapidly Tuning the PID Controller Based on the Regional Surrogate Model Technique in the UAV Formation

Wang

Duan

Yan

et al. 2020

Entropy

View full text Add to dashboard Cite

The leader–follower structure is widely used in unmanned aerial vehicle formation. This paper adopts the proportional-integral-derivative (PID) and the linear quadratic regulator controllers to construct the leader–follower formation. Tuning the PID controllers is generally empirical; hence, various surrogate models have been introduced to identify more refined parameters with relatively lower cost. However, the construction of surrogate models faces the problem that the singular points may affect the accuracy, such that the global surrogate models may be invalid. Thus, to tune controllers quickly and accurately, the regional surrogate model technique (RSMT), based on analyzing the regional information entropy, is proposed. The proposed RSMT cooperates only with the successful samples to mitigate the effect of singular points along with a classifier screening failed samples. Implementing the RSMT with various kinds of surrogate models, this study evaluates the Pareto fronts of the original simulation model and the RSMT to compare their effectiveness. The results show that the RSMT can accurately reconstruct the simulation model. Compared with the global surrogate models, the RSMT reduces the run time of tuning PID controllers by one order of magnitude, and it improves the accuracy of surrogate models by dozens of orders of magnitude.

show abstract

Information-Theoretic Scores for Bayesian Model Selection and Similarity Analysis: Concept and Application to a Groundwater Problem

Oreamuno

Oladyshkin

Nowak

2022

Preprint

View full text Add to dashboard Cite

Bayesian model selection (BMS) and Bayesian model justifiability analysis (BMJ) provide a statistically rigorous framework to compare competing conceptual models through the use of Bayesian model evidence (BME). However, BME-based analysis has two main limitations: (1) it's powerless when comparing models with different data set sizes and/or types of data and(2) doesn't allow to judge a model's performance based on its posterior predictive capabilities. Thus, traditional BME-based approaches ignore useful data or models due to issue (1) or disregards Bayesian updating because of issue (2). To address these limitations, we advocate to include additional information-theoretic scores into BMS and BMJ analysis: expected log-predictive density (ELPD), relative entropy (RE) and information entropy (IE). Exploring the connection between Bayesian inference and information theory, we explicitly link BME and ELPD together with RE and IE to indicate the information flow in BMS and BMJ analysis. We show how to compute and interpret these scores alongside BME, and apply it in a model selection and similarity analysis framework. We test the methodology on a controlled 2D groundwater setup considering five competing conceptual models accompanied with different data sets. The results show how the information-theoretic scores complement BME by providing a more complete picture concerning the Bayesian updating process. Additionally, we present how both RE and IE can be used to objectively compare models that feature different data sets. Overall, the introduced Bayesian information-theoretic framework helps to avoid any potential loss of information and leads to an informed decision for model selection and similarity.

show abstract

The Connection between Bayesian Inference and Information Theory for Model Selection, Information Gain and Experimental Design

Cited by 21 publications

References 54 publications

Bayesian3 Active Learning for the Gaussian Process Emulator Using Information Theory

Bayesian3 Active Learning for the Gaussian Process Emulator Using Information Theory

Rapidly Tuning the PID Controller Based on the Regional Surrogate Model Technique in the UAV Formation

Information-Theoretic Scores for Bayesian Model Selection and Similarity Analysis: Concept and Application to a Groundwater Problem

Contact Info

Product

Resources

About