Analyzing Stochastic Computer Models: A Review with Opportunities

Abstract

In modern science, computer models are often used to understand complex phenomena, and a thriving statistical community has grown around analyzing them. This review aims to bring a spotlight to the growing prevalence of stochastic computer models -providing a catalogue of statistical methods for practitioners, an introductory view for statisticians (whether familiar with deterministic computer models or not), and an emphasis on open questions of relevance to practitioners and statisticians. Gaussian process surrogate models take center stage in this review, and these, along with several extensions needed for stochastic settings, are explained. The basic issues of designing a stochastic computer experiment and calibrating a stochastic computer model are prominent in the discussion. Instructive examples, with data and code, are used to describe the implementation of, and results from, various methods.

Batch-sequential design and heteroskedastic surrogate modeling for delta smelt conservation

“…Kennedy and O'Hagan (2001)-style calibration of stochastic simulators remains on the frontier of design for surrogate modeling. Baker et al (2020) identify this as an important area for further research.…”

“…In our initial study, described in Section 2.3, we observe that the response surface is nonlinear and heteroskedastic, i.e., sensitivity to stochastic simulation dynamics is not uniform in the input space. These features challenge effective design and meta-modeling, which are essential for downstream tasks like input sensitivity analysis and calibration, mirroring ones which are increasingly common the analysis of stochastic simulation experiments (Baker et al, 2020).…”

libEnsemble: A Library to Coordinate the Concurrent Evaluation of Dynamic Ensembles of Calculations

“…however, determining parameters to evaluate dynamically can lead to better emulator properties. Focusing evaluations on portions of a parameter space based on information obtained from previous evaluations can significantly improve emulator quality in parameter regions of particular interest for model calibration [29][30][31].…”

VECMAtk: a scalable verification, validation and uncertainty quantification toolkit for scientific simulations

“…Collectively, the processes involved in evaluating our level of trust in the results obtained from models are known as VVUQ. VVUQ processes provide the basis for determining our level of trust in any given model and the results obtained using it [1,58,59].…”