Testing for ontological errors in probabilistic forecasting models of natural systems

Abstract

Probabilistic forecasting models describe the aleatory variability of natural systems as well as our epistemic uncertainty about how the systems work. Testing a model against observations exposes ontological errors in the representation of a system and its uncertainties. We clarify several conceptual issues regarding the testing of probabilistic forecasting models for ontological errors: the ambiguity of the aleatory/epistemic dichotomy, the quantification of uncertainties as degrees of belief, the interplay between Bayesian and frequentist methods, and the scientific pathway for capturing predictability. We show that testability of the ontological null hypothesis derives from an experimental concept, external to the model, that identifies collections of data, observed and not yet observed, that are judged to be exchangeable when conditioned on a set of explanatory variables. These conditional exchangeability judgments specify observations with well-defined frequencies. Any model predicting these behaviors can thus be tested for ontological error by frequentist methods; e.g., using P values. In the forecasting problem, prior predictive model checking, rather than posterior predictive checking, is desirable because it provides more severe tests. We illustrate experimental concepts using examples from probabilistic seismic hazard analysis. Severe testing of a model under an appropriate set of experimental concepts is the key to model validation, in which we seek to know whether a model replicates the data-generating process well enough to be sufficiently reliable for some useful purpose, such as long-term seismic forecasting. Pessimistic views of system predictability fail to recognize the power of this methodology in separating predictable behaviors from those that are not.system science | Bayesian statistics | significance testing | subjective probability | expert opinion S cience is rooted in the concept that a model can be tested against observations and rejected when necessary (1). However, the problem of model testing becomes formidable when we consider natural systems. Owing to their scale, complexity, and openness to interactions within a larger environment, most natural systems cannot be replicated in the laboratory, and direct observations of their inner workings are always inadequate. These difficulties raise serious questions about the meaning and feasibility of "model verification" and "model validation" (2), and have led to the pessimistic view that "the outcome of natural processes in general cannot be accurately predicted by mathematical models" (3).Uncertainties in the formal representation of natural systems imply that the forecasting of emergent phenomena such as natural hazards must be based on probabilistic rather than deterministic modeling. The ontological framework for most probabilistic forecasting models comprises two types of uncertainty: an aleatory variability that describes the randomness of the system, and an epistemic uncertainty that characterizes our lack of knowledge about the sy...

The effect of uncertainty in earthquake fault parameters on the maximum wave height from a tsunami propagation model

“…While in some cases this may be more computationally efficient, it may not be strictly correct according to the normal definition of those terms. Epistemic uncertainty, as defined by for example Marzocchi and Jordan (2014), is due to our lack of knowledge of the system, while aleatory uncertainty (or variability) is due to the intrinsic randomness of the system. In theory, epistemic uncertainty can be reduced by more knowledge about the system, but aleatory uncertainty cannot.…”

Accounting for Epistemic Uncertainty in PSHA: Logic Tree and Ensemble Modeling

“…Indeed, this discussion is deeply rooted in the intrinsic meaning of probability (frequency versus degree of belief) and, more importantly, in the possibility of validating a probabilistic assessment like the outcome of probabilistic seismic-hazard analysis (PSHA). Marzocchi and Jordan (2014) suggest that a clear and univocal taxonomy of uncertainties is not only of practical convenience, but it is of primary importance to meaningfully validate any probabilistic assessment and, consequently, to keep PSHA in a scientific domain (see Marzocchi and Jordan, 2014, for a discussion on commonalities and differences with the traditional view of PSHA practitioners; as exemplified in SSHAC, 1997). In particular, Marzocchi and Jordan (2014) show that aleatory variability and epistemic uncertainty can be separated only in the framework of a well-defined experimental concept.…”

A Comparison between the Forecast by the United States National Seismic Hazard Maps with Recent Ground‐Motion Records

“…1a,b), testing an ensemble forecast requires computing the forecast as the weighted sum of distribution functions, based on the law of total probability. As pointed out by Marzocchi and Jordan (2014), the result will be generally different from that based only on the mean hazard curve. When the abscissa of the aggregated hazard curve is expressed in return periods (e.g., Fig.…”