The Four Ways to Consider Measurement Noise in Bayesian Model Selection—And Which One to Choose

Abstract

Models are used to predict and/or investigate and explain phenomena in nature. Often, many hypotheses exist for these two tasks. Naturally, the question arises, which of the competing modeling approaches predicts or explains nature best. Bayesian model selection (BMS, e.g., Wasserman, 2000) is a statistical method that uses observed data to select between competing models. BMS is settled in a rigorous probabilistic framework and follows the scheme of Bayesian updating: A prior belief about the plausibility of each candidate model is updated to a posterior model weight in the light of measured data (i.e., the probability of the model to have generated the data, given the model set). Posterior model weights are then used as a basis for Bayesian model ranking, selection, or averaging (BMA, Hoeting et al., 1999).To help with the interpretation of posterior model weights, the so-called model confusion matrix (MCM) has been introduced by Schöniger, Illman, et al. (2015). It reveals whether a lack of confidence in model choice is due to similarity between the candidate models or due to weakly informative data. The MCM is a purely synthetic analysis that can be used as a scale of reference for model weights obtained from real data. Schäfer Rodrigues Silva et al. (2020) have recently extended the MCM analysis to identify the best surrogate model from a set of candidates to replace an expensive full-complexity model in stochastic analysis.Technically, the Bayesian updating procedure requires calculating the so-called Bayesian model evidence (BME). BME is the likelihood of a model to have generated the data, integrated over its whole parameter space and all involved probability distributions. While the likelihood accounts for uncertainty in measured data, the integration considers parameter uncertainty, and potentially also uncertainty in model drivers or boundary conditions. In some cases, the integration even accounts for statistical representations of model errors (Leube et al., 2012;Nowak et al., 2012), which is perceived by many studies to be part of the likelihood.

The Treatment of Uncertainty in Hydrometric Observations: A Probabilistic Description of Streamflow Records

“…Discharge records are subject to uncertainty and its estimation can reduce economic costs of water management decisions (H. McMillan et al., 2017) and allows robust research conclusions to be drawn, as demonstrated in data assimilation (Burgers et al., 1998), regionalization (Westerberg et al., 2016), and Bayesian model selection (Reuschen et al., 2021). Discharge time series are usually obtained by relating continuously measured river stage to discharge through so‐called rating curves.…”

Multipoint Deformation Safety Monitoring Model for Concrete Arch Dams Based on Bayesian Model Selection and Averaging

“…Te posterior probabilities of each model can be used as the weights for inference, and the weighted average of the models of interest can be obtained to determine the BMA [39][40][41][42][43]. Te BMA modeling approach is based on a large model space, but performing scientifc sampling in the model space to identify models with high posterior probabilities remains a challenge in applying BMA.…”

Information-Theoretic Scores for Bayesian Model Selection and Similarity Analysis: Concept and Application to a Groundwater Problem

“…In BMJ, N d realizations from the parameter prior of each data-generating model M j are sampled and evaluated in the model. Noise is then added to each data set to account for the measurement error associated to real observations (Reuschen et al, 2021).…”

“…If one had an infinite number of model realizations, the geostatistical model would be able to reproduce data generated from itself perfectly. To properly account for measurement noise in this synthetic setup for BMS and BMJ analysis, noise was added to the synthetic data set, to account for measurement error (Reuschen et al, 2021). For the noise, we consider a standard deviation of h error = 0.06m and c error = 0.06 + 0.2c o ,…”

“…It can therefore complement the model selection analysis. Recently, the BMJ framework has been extended to computationally demanding models applying surrogates (Schäfer Rodrigues Silva et al, 2020;Scheurer et al, 2021) and for model uncertainty quantification (Reuschen et al, 2021).…”