Replication or Exploration? Sequential Design for Stochastic Simulation Experiments

Abstract

We investigate the merits of replication, and provide methods for optimal design (including replicates), with the goal of obtaining globally accurate emulation of noisy computer simulation experiments. We first show that replication can be beneficial from both design and computational perspectives, in the context of Gaussian process surrogate modeling. We then develop a lookahead based sequential design scheme that can determine if a new run should be at an existing input location (i.e., replicate) or at a new one (explore). When paired with a newly developed heteroskedastic Gaussian process model, our dynamic design scheme facilitates learning of signal and noise relationships which can vary throughout the input space. We show that it does so efficiently, on both computational and statistical grounds. In addition to illustrative synthetic examples, we demonstrate performance on two challenging real-data simulation experiments, from inventory management and epidemiology.

Predicting the Output From a Stochastic Computer Model When a Deterministic Approximation is Available

“…Another, potentially promising, idea for design would be the inclusion of replicated simulator runs in the fitting of the emulator. Replicates are often used in the fitting of heteroscedastic emulators (Ankenman et al, 2010;Binois et al, 2017), with the goal of obtaining a better understanding of the simulator's mean and variance for the observed input points. With DetHetGP substantially improving the mean of stochastic emulator predictions, at the expense of fewer stochastic simulator runs, and sometimes at the expense of a worse variance prediction, it would be interesting to see whether combining replicates with incorporating deterministic approximations would yield an improved emulator overall.…”

“…Emulator predictions for the toy simulator fromBinois et al (2017), using the newly developed model that incorporates both stochastic and deterministic runs. The True mean and 95% interval are superimposed in black, and the emulator mean and 95% interval are in blue.…”

Distance-Distributed Design for Gaussian Process Surrogates

“…Considering nugget hyperparameters in the optimization would add yet another layer of complication. In that setting, we may wish to consider replication (i.e., zero-inflated distance distributions) as a means of separating signal from noise (Binois et al, 2018b).…”

A Stochastic-Kriging-Based Multiobjective Simulation Optimization Algorithm

“…This obviously is problem-specific, but the merits of replicating can be further studied to develop strategies that foresee the impact of either replicating a smaller number of already sampled points or investing more budget exploring the design space. A very recent attempt to approach this problem appears in [2], but a great amount of work remains to be done in this regard.…”