Approximate Laplace importance sampling for the estimation of expected Shannon information gain in high-dimensional Bayesian design for nonlinear models

Abstract

One of the major challenges in Bayesian optimal design is to approximate the expected utility function in an accurate and computationally efficient manner. We focus on Shannon information gain, one of the most widely used utilities when the experimental goal is parameter inference. We compare the performance of various methods for approximating expected Shannon information gain in common nonlinear models from the statistics literature, with a particular emphasis on Laplace importance sampling (LIS) and approximate Laplace importance sampling (ALIS), a new method that aims to reduce the computational cost of LIS. Specifically, in order to centre the importance distributions LIS requires computation of the posterior mode for each of a large number of simulated possibilities for the response vector. ALIS substantially reduces the amount of numerical optimization that is required, in some cases eliminating all optimization, by centering the importance distributions on the data-generating parameter values wherever possible. Both methods are thoroughly compared with existing approximations including Double Loop Monte Carlo, nested importance sampling, and Laplace approximation. It is found that LIS and ALIS both give an efficient trade-off between mean squared error and computational cost for utility estimation, and ALIS can be up to 70% cheaper than LIS. Usually ALIS gives an approximation that is cheaper but less accurate than LIS, while still being efficient, giving a useful addition to the suite of efficient methods. However, we observed one case where ALIS is both cheaper and more accurate. In addition, for the first time we show that LIS and ALIS yield superior designs to existing methods in problems with large numbers of model parameters when combined with the approximate co-ordinate exchange algorithm for design optimization.

Variational Experimental Design Methods for Geophysical Applications

“…Therefore they should introduce a similar bias as incurred when using a well-trained MDN predicting one Gaussian with full covariance matrix. The same goes for the consistent extensions of both methods, the VNMC method of Foster et al (2019a) and the Laplace-based importance sampling estimator of Carlon et al (2020); Englezou et al (2022). Unlike the variational methods, the Laplace-based methods are inherently restricted to a Gaussian posterior pdf.…”

“…While functional approximations introduce additional complexity compared to straightforward double-loop Monte Carlo estimators such as the NMC method, they have significant advantages, especially since the NMC method with reused samples, while working well in the presented examples, can perform suboptimal when compared to methods using functional approximations (Englezou et al, 2022). Most importantly, they allow the design of experiments best suited to answer any scientific or applied question, provided a mapping from model space to the relevant target space can be defined.…”

Enhancing LAN Failure Predictions with Decision Trees and SVMs: Methodology and Implementation

“…The first step in building a decision tree is to determine the best attribute to split the data. This is typically done by using a measure of impurity like the Gini index [26,27] or information gain [27,28]. For example, information gain can be calculated as follows:…”