Statistical Tests for Cross-Validation of Kriging Models

Abstract

Kriging or Gaussian process models are popular metamodels (surrogate models or emulators) of simulation models; these metamodels give predictors for input combinations that are not simulated. To validate these metamodels for computationally expensive simulation models, the analysts often apply computationally efficient cross-validation. In this paper, we derive new statistical tests for so-called leave-one-out cross-validation. Graphically, we present these tests as scatterplots augmented with confidence intervals that use the estimated variances of the Kriging predictors. To estimate the true variances of these predictors, we might use bootstrapping. Like other statistical tests, our tests—with or without bootstrapping—have type I and type II error probabilities; to estimate these probabilities, we use Monte Carlo experiments. We also use such experiments to investigate statistical convergence. To illustrate the application of our tests, we use (i) an example with two inputs and (ii) the popular borehole example with eight inputs. Summary of Contribution: Simulation models are very popular in operations research (OR) and are also known as computer simulations or computer experiments. A popular topic is design and analysis of computer experiments. This paper focuses on Kriging methods and cross-validation methods applied to simulation models; these methods and models are often applied in OR. More specifically, the paper provides the following; (1) the basic variant of a new statistical test for leave-one–out cross-validation; (2) a bootstrap method for the estimation of the true variance of the Kriging predictor; and (3) Monte Carlo experiments for the evaluation of the consistency of the Kriging predictor, the convergence of the Studentized prediction error to the standard normal variable, and the convergence of the expected experimentwise type I error rate to the prespecified nominal value. The new statistical test is illustrated through examples, including the popular borehole model.

Sequential metamodel‐based approaches to level‐set estimation under heteroscedasticity

“…There exist several methods for constructing a confidence bound for the unknown mean response function, including the classical pointwise confidence interval [34], the simultaneous confidence region relying on bootstrapping or the Bonferroni [24] and Šidák corrections [13], and the uniform confidence bounds derived either using the frequentist kernel methods [23] or from the Bayesian GP modeling perspective [43]. We adopt the uniform bound for heteroscedastic metamodeling approaches (including SK) proposed by Kirschner and Krause [23] which holds true with a prescribed high probability across the input space and through all iterations $$$ t\ge 1 $$$.…”

Research on Emergency Rescue Response in Wenchuan County in China Based on Baidu Map Navigation Data

“…To get the distribution of emergency rescue response level in the study area, the normalized data need to be quantified spatially. This study uses the Kriging interpolation method for spatial quantization (Kleijnen and van Beers, 2022;Meng, 2021). This interpolation method has the following two advantages:…”