Making Steppingstones out of Stumbling Blocks: A Bayesian Model Evidence Estimator with Application to Groundwater Transport Model Selection

Abstract

Bayesian model evidence (BME) is a measure of the average fit of a model to observation data given all the parameter values that the model can assume. By accounting for the trade-off between goodness-of-fit and model complexity, BME is used for model selection and model averaging purposes. For strict Bayesian computation, the theoretically unbiased Monte Carlo based numerical estimators are preferred over semi-analytical solutions. This study examines five BME numerical estimators and asks how accurate estimation of the BME is important for penalizing model complexity. The limiting cases for numerical BME estimators are the prior sampling arithmetic mean estimator (AM) and the posterior sampling harmonic mean (HM) estimator, which are straightforward to implement, yet they result in underestimation and overestimation, respectively. We also consider the path sampling methods of thermodynamic integration (TI) and steppingstone sampling (SS) that sample multiple intermediate distributions that link the prior and the posterior. Although TI and SS are theoretically unbiased estimators, they could have a bias in practice arising from numerical implementation. For example, sampling errors of some intermediate distributions can introduce bias. We propose a variant of SS, namely the multiple one-steppingstone sampling (MOSS) that is less sensitive to sampling errors. We evaluate these five estimators using a groundwater transport model selection problem. SS and MOSS give the least biased BME estimation at an efficient computational cost. If the estimated BME has a bias that covariates with the true BME, this would not be a problem because we are interested in BME ratios and not their absolute values. On the contrary, the results show that BME estimation bias can be a function of model complexity. Thus, biased BME estimation results in inaccurate penalization of more complex models, which changes the model ranking. This was less observed with SS and MOSS as with the three other methods.

Process Interactions Can Change Process Ranking in a Coupled Complex System Under Process Model and Parametric Uncertainty

“…Redefining the first-order and total-effect process sensitivity indices using the posterior distributions and weights is straightforward, as discussed in . The key challenge is to estimate the posterior parameter distributions and posterior process weights due to high computational cost of Markov chain Monte Carlo simulations (Elshall and Ye, 2019;Liu et al, 2016). Estimating the posterior process weights are particularly challenging because it is different from estimating posterior system model weights and has not been attempted.…”

The Four Ways to Consider Measurement Noise in Bayesian Model Selection—And Which One to Choose

“…As argued by Schöniger et al (2014), Monte Carlo is superior to other numerical schemes in that it is an unbiased scheme that is known to converge to the correct limit, and its convergence can be easily monitored. In our chosen test cases, the computational burden of Monte Carlo is bearable; for computationally heavier practical applications, alternative numerical methods could be used to improve on computational efficiency, such as nested sampling (Elsheikh et al, 2014;Skilling, 2006), thermodynamic integration (Lartillot & Philippe, 2006;Liu et al, 2016), stepping stone sampling (Elshall & Ye, 2019;Xie et al, 2011), or Gaussian mixture importance sampling (Volpi et al, 2017), to name a few examples. However, these methods are less straightforward to implement and bear the risk of introducing biases into the BME estimation.…”

“…Due to its statistical rigor and its elegance in accounting for uncertainty, BMS has become popular in water resources research. It has been applied in various different contexts, such as evaluation of hydrological models (Marshall et al., 2005), frequency analysis of hydrological extremes (Laio et al., 2009), climate change impact studies (Najafi et al., 2011), model complexity analysis (Höge et al., 2018; Schöniger, Illman, et al., 2015), optimal design for model choice (Nowak & Guthke, 2016), as well as hydrogeophysical (Brunetti et al., 2017), hydro‐morphodynamic (Mohammadi et al., 2018), and groundwater transport modeling (Elshall & Ye, 2019).…”

“…cal models (Marshall et al, 2005), frequency analysis of hydrological extremes (Laio et al, 2009), climate change impact studies (Najafi et al, 2011), model complexity analysis (Höge et al, 2018;Schöniger, Illman, et al, 2015), optimal design for model choice (Nowak & Guthke, 2016), as well as hydrogeophysical (Brunetti et al, 2017), hydro-morphodynamic (Mohammadi et al, 2018), and groundwater transport modeling (Elshall & Ye, 2019).…”

Information-Theoretic Scores for Bayesian Model Selection and Similarity Analysis: Concept and Application to a Groundwater Problem

“…Traditionally, model ranking in the BMS framework is based on the values of Bayesian model evidence (BME), which are defined as the probability of a model of reproducing the available data (Raftery, 1995;Draper, 1995). Such BME-based model selection approaches have been used in many fields for model ranking, and/or selection purposes, for example: Schöniger, Illman, et al (2015) and Elshall and Ye (2019) for groundwater modelling, Wöhling et al (2015) for crop modelling, Marshall et al (2005) for hydrological models, Brunetti et al (2017) in hydrogeophysical modelling and Schäfer Rodrigues Silva et al (2020) in reactive groundwater transport models, to name a few. Additionally, Mohammadi et al (2018) and Scheurer et al (2021) apply BMS using surrogate models for sediment transport in rivers and to biochemical processes in the subsurface, respectively.…”