A comparison of statistical emulation methodologies for multi‐wave calibration of environmental models

Abstract

Expensive computer codes, particularly those used for simulating environmental or geological processes, such as climate models, require calibration (sometimes called tuning). When calibrating expensive simulators using uncertainty quantification methods, it is usually necessary to use a statistical model called an emulator in place of the computer code when running the calibration algorithm. Though emulators based on Gaussian processes are typically many orders of magnitude faster to evaluate than the simulator they mimic, many applications have sought to speed up the computations by using regression‐only emulators within the calculations instead, arguing that the extra sophistication brought using the Gaussian process is not worth the extra computational power. This was the case for the analysis that produced the UK climate projections in 2009. In this paper, we compare the effectiveness of both emulation approaches upon a multi‐wave calibration framework that is becoming popular in the climate modeling community called “history matching.” We find that Gaussian processes offer significant benefits to the reduction of parametric uncertainty over regression‐only approaches. We find that in a multi‐wave experiment, a combination of regression‐only emulators initially, followed by Gaussian process emulators for refocussing experiments can be nearly as effective as using Gaussian processes throughout for a fraction of the computational cost. We also discover a number of design and emulator‐dependent features of the multi‐wave history matching approach that can cause apparent, yet premature, convergence of our estimates of parametric uncertainty. We compare these approaches to calibration in idealized examples and apply it to a well‐known geological reservoir model.

The importance of comprehensive parameter sampling and multiple observations for robust constraint of aerosol radiative forcing

“…10). Furthermore, the likely magnitude of forcing derived from emergent constraints is sensitive to the uncertainties accounted for in the process (Samset et al, 2014).…”

Uncertainty Quantification for Computer Models With Spatial Output Using Calibration-Optimal Bases

“…There are many variants on emulation, with some practitioners preferring no regressors (Chen et al, 2016), different types of correlation function (including no correlation) (Kaufman et al, 2011;Salter and Williamson, 2016), and different priors, π(β, φ), with some leading to partially analytic posterior inference (Haylock and O'Hagan, 1996). As history matching only requires posterior means and variances of the emulator, Bayes linear analogues are sometimes used (Vernon et al, 2010).…”

“…Our illustration of the terminal case shows that though careful subjective prior information is required for model discrepancy in order to overcome the identifiability issues with the calibration model, if those judgements lead to a prior-data conflict via a terminal case, good calibration will not be possible, and it will take a great deal of resource (enough data to build a near perfect emulator everywhere) to discover this. It would seem more natural to first history match in order to check we are not in a terminal case, and, if not, perform a probabilistic calibration within NROY space as in Salter and Williamson (2016).…”

Process‐Based Climate Model Development Harnessing Machine Learning: II. Model Calibration From Single Column to Global

“…We note that, though we have performed 20 waves, here, the objective is not to find a single good simulation, which could be done using a Bayesian procedure within NROY space (Salter & Williamson, 2016), but to identify all good matches in order to use this subspace for the tuning of the 3D GCM.…”