Model selection on solid ground: Rigorous comparison of nine ways to evaluate Bayesian model evidence

Abstract

Bayesian model selection or averaging objectively ranks a number of plausible, competing conceptual models based on Bayes' theorem. It implicitly performs an optimal trade-off between performance in fitting available data and minimum model complexity. The procedure requires determining Bayesian model evidence (BME), which is the likelihood of the observed data integrated over each model's parameter space. The computation of this integral is highly challenging because it is as high-dimensional as the number of model parameters. Three classes of techniques to compute BME are available, each with its own challenges and limitations: (1) Exact and fast analytical solutions are limited by strong assumptions. (2) Numerical evaluation quickly becomes unfeasible for expensive models. (3) Approximations known as information criteria (ICs) such as the AIC, BIC, or KIC (Akaike, Bayesian, or Kashyap information criterion, respectively) yield contradicting results with regard to model ranking. Our study features a theory-based intercomparison of these techniques. We further assess their accuracy in a simplistic synthetic example where for some scenarios an exact analytical solution exists. In more challenging scenarios, we use a brute-force Monte Carlo integration method as reference. We continue this analysis with a real-world application of hydrological model selection. This is a first-time benchmarking of the various methods for BME evaluation against true solutions. Results show that BME values from ICs are often heavily biased and that the choice of approximation method substantially influences the accuracy of model ranking. For reliable model selection, bias-free numerical methods should be preferred over ICs whenever computationally feasible.

Bayesian model selection in hydrogeophysics: Application to conceptual subsurface models of the South Oyster Bacterial Transport Site, Virginia, USA

“…(3) ), computed with respect to each tested conceptual models, are all larger than 10 100 . This result is in agreement with the findings by Schöniger et al (2014) : one decisive winning conceptual model is often obtained when using large data sets and small data errors. We also considered the field example described in Section 4.1 , but using less data (i.e., n = 224 instead of n = 3248 ) and we found (results not shown) that: (1) the isotropic multiGaussian model is still the winner, (2) all the evidence estimates are much larger (e.g., in the case of the isotropic multi-Gaussian model, the evidence increases from about 10 −10 0 0 to 10 −100 ) and that (3) the Bayes factors are much smaller (e.g., when comparing the multi-Gaussian model with vertical anisotropy and the one with isotropy, the Bayes factor decreases approximately from 10 190 to 10 10 ).…”

“…For our set-up with small errors and high data and model dimensions, we found that reliable evidence estimation with the BFMC method would need prohibitive computation times. If the assumption of a multi-Gaussian posterior density is fulfilled (a reasonable assumption in our test cases), the LM method should provide reliable evidence estimates (see also case-studies by Schöniger et al (2014) ). This is confirmed in our synthetic study in Section 3 by the strong agreement at low model dimensions between BFMC and LM estimates evaluated around the MAP estimate.…”

“…A recent study by Schöniger et al (2014) elucidates that AIC and BIC do a rather poor job in ranking hydrologic models. The authors of this study therefore concluded that AIC and BIC are a relatively poor proxy of the evidence.…”

“…What is more, the brute force Monte Carlo method is known to be affected by the so-called curse of dimensionality which degenerates the evidence estimates and make them unusable in high dimensions ( Lewis and Raftery, 1997 ). In cases where reliable brute force Monte Carlo integration is infeasible, Schöniger et al (2014) promote the use of KIC for model selection, evaluated at the maximum a-posteriori (MAP) density parameter values of the posterior distribution. Note that the KIC is a simple transform of evidence estimates obtained by the LaplaceMetropolis method ( Lewis and Raftery, 1997 ).…”

Bayesian sparse polynomial chaos expansion for global sensitivity analysis

“…BMA, relying on Bayes' theorem, is a well known statistical approach to perform quantitative comparisons of competing models [16,17]. The difficulty of BMA lies in the evaluation of a quantity referred to as the 'Bayesian model evidence' (BME), which involves an integral over the whole input space, so it generally has no analytical expression.…”