Assessment and Propagation of Model Uncertainty

Abstract

SUMMARY In most examples of inference and prediction, the expression of uncertainty about unknown quantities y on the basis of known quantities x is based on a model M that formalizes assumptions about how x and y are related. M will typically have two parts: structural assumptions S, such as the form of the link function and the choice of error distribution in a generalized linear model, and parameters θ whose meaning is specific to a given choice of S. It is common in statistical theory and practice to acknowledge parametric uncertainty about θ given a particular assumed structure S; it is less common to acknowledge structural uncertainty about S itself. A widely used approach involves enlisting the aid of x to specify a plausible single ‘best’ choice S* for S, and then proceeding as if S* were known to be correct. In general this approach fails to assess and propagate structural uncertainty fully and may lead to miscalibrated uncertainty assessments about y given x. When miscalibration occurs it will often result in understatement of inferential or predictive uncertainty about y, leading to inaccurate scientific summaries and overconfident decisions that do not incorporate sufficient hedging against uncertainty. In this paper I discuss a Bayesian approach to solving this problem that has long been available in principle but is only now becoming routinely feasible, by virtue of recent computational advances, and examine its implementation in examples that involve forecasting the price of oil and estimating the chance of catastrophic failure of the US space shuttle.

Bayesian inference for categorical data analysis

“…It accounts for uncertainty about the model by taking an average of the posterior distribution of a quantity of interest, weighted by the posterior probabilities of several potential models. Following the previously discussed work of Madigan and Raftery (1994), the idea of model averaging was developed further by Draper (1995) and Raftery, Madigan, and Hoeting (1997). In their review article, Hoeting et al (1999) discussed model averaging in the context of GLMs.…”

A strategy for modelling the effect of a continuous covariate in medicine and epidemiology

“…The de"ciency becomes apparent when data from new studies are examined and the original model is found to be seriously biased. Several authors (for example, Draper [24]) have recognized and suggested ways of trying to overcome a related problem, that of model uncertainty; statistical inference is routinely applied to the parameters as though the model were the true and only one, without taking into account many others that were tried and rejected during data exploration. Since data analysis and the selection of a model are complex, ill-de"ned and often highly subjective processes, it is unlikely that any satisfactory procedure will ever be found to accommodate model uncertainty adequately.…”

Incorporating subjective and stochastic uncertainty in an interactive multi-objective groundwater calibration framework

“…This approach, called Bayesian model averaging-or more specifically maximum likelihood Bayesian model averaging (MLBMA), in the case of Neuman (2003)-uses a more statistically consistent methodology to assess the Bayesian posterior probabilities for a given conceptual model. While approaches such as Draper (1995), Kass and Raftery (1995), and Hoeting et al (1999) rely on extensive Monte Carlo simulations to calculate the probabilities, Neuman (2003) proposed using likelihood measures such as Kashyap information criterion (KIC;Kashyap 1982) and Bayesian information criterion (BIC; Schwarz 1978) to weight different models, thus obviating the need for extensive simulations.…”

“…The other major framework for addressing model uncertainty is the formal Bayesian approach, which has been used by Draper (1995), Kass and Raftery (1995), Hoeting et al (1999), Woodbury and Ulrych (2000), Neuman (2003), Neuman and Wierenga (2003), and Ye et al (2004), among others. This approach, called Bayesian model averaging-or more specifically maximum likelihood Bayesian model averaging (MLBMA), in the case of Neuman (2003)-uses a more statistically consistent methodology to assess the Bayesian posterior probabilities for a given conceptual model.…”