Monotonicity-Preserving Bootstrapped Kriging Metamodels for Expensive Simulations

Kleijnen, J.P.C.; Beers, W.C.M. van

doi:10.2139/ssrn.1477896

Cited by 20 publications

(29 citation statements)

References 32 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Paired t-test are carried out and the results suggest that the difference between the coverage for the MNEK and BK is statistically significant at an alpha level of 0.05. We note that the coverage falls short of the stated level at high traffic level as also observe in Kleijnen and Beers (2010). As for the predictive variance, we see from Fig.…”

Section: Numerical Examplesupporting

confidence: 67%

A Bayesian metamodeling approach for stochastic simulations

Yin¹,

Ng²,

Ng³

2010

Proceedings of the 2010 Winter Simulation Conference

View full text Add to dashboard Cite

In the application of kriging model in the field of simulation, the parameters of the model are likely to be estimated from the simulated data. This introduces parameter estimation uncertainties into the overall prediction error, and this uncertainty can be further aggravated by random noise in the stochastic simulation. In this paper, a Bayesian metamodeling approach for kriging prediction is proposed for stochastic simulations to more appropriately account for the parameter uncertainties. The approach is first illustrated analytically using a simplified two point example. A more general Markov Chain Monte Carlo analysis approach is subsequently proposed to handle more general assumptions on the parameters and design. The general MCMC approach is compared with the modified nugget effect kriging model based on the M/M/1 simulation system. Initial results indicate that the Bayesian approach has better coverage and closer predictive variance to the empirical value than the modified nugget effect kriging model, especially in the cases where the stochastic variability is high.

show abstract

Section: Numerical Examplesupporting

confidence: 67%

A Bayesian metamodeling approach for stochastic simulations

Yin¹,

Ng²,

Ng³

2010

Proceedings of the 2010 Winter Simulation Conference

View full text Add to dashboard Cite

show abstract

“…Kleijnen and van Beers (2011) find that w r and w .9;r are not normally distributed if the simulation run is as short as T = 1000, even for the relatively low traffic rate 0.5.…”

Section: Monotonicity: M/m/1 Queue Simulationmentioning

confidence: 85%

“…To further examine this low coverage, Kleijnen and van Beers (2011) increases n from 5 to 10. This change increases the estimated coverages for both classic and monotonic Kriging; this improved coverage may be explained by the better fit of the Kriging model resulting from an "adequate" sample size; also see Loeppky, Sacks, and Welch (2009), suggesting that a valid Kriging metamodel requires n = 10k (which in the M/M/1 example implies n = 10).…”

Section: Kleijnen Mehdad and Van Beersmentioning

confidence: 99%

“…This Kriging model implies estimates ∂ C/∂s and ∂ C/∂Q at a specific point (say) (s 0 , S 0 ); see again (2). This metamodel is less precise at interpolated new points than it is at simulated old points; nevertheless, we compute not only the predicted first-order derivatives at the n old points, but also at 10000 new points on a 100 × 100 grid (in their M/M/1 simulation Kleijnen and van Beers (2011) also estimate derivatives at new points). These first-order derivatives imply the following estimates of the second-order derivatives at the point (s 0 , Q 0 ), given the old points i (i =1, ..., n):…”

Section: Kleijnen Mehdad and Van Beersmentioning

confidence: 99%

See 1 more Smart Citation

Convex and monotonic bootstrapped Kriging

Kleijnen

Mehdad

Beers

2012

Proceedings Title: Proceedings of the 2012 Winter Simulation Conference (WSC)

Self Cite

View full text Add to dashboard Cite

Distribution-free bootstrapping of the replicated responses of a given discrete-event simulation model gives bootstrapped Kriging (Gaussian process) metamodels; we require these metamodels to be either convex or monotonic. To illustrate monotonic Kriging, we use an M/M/1 queueing simulation with as output either the mean or the 90% quantile of the transient-state waiting times, and as input the traffic rate. In this example, monotonic bootstrapped Kriging enables better sensitivity analysis than classic Kriging; i.e., bootstrapping gives lower MSE and confidence intervals with higher coverage and the same length. To illustrate convex Kriging, we start with simulation-optimization of an (s, S) inventory model, but we next switch to a Monte Carlo experiment with a second-order polynomial inspired by this inventory simulation. We could not find truly convex Kriging metamodels, either classic or bootstrapped; nevertheless, our bootstrapped "nearly convex" Kriging does give a confidence interval for the optimal input combination. INTRODUCTIONMany realistic simulation models have known characteristics such as convexity and monotonicity. For example, simulation models of supply chains consist of a sequence of submodels (building blocks, modules) for queues and inventories; higher traffic rates monotonically increase mean waiting time, and reorder levels and order quantities are often assumed to have a unique optimal combination because the cost function is convex (instead of having multiple local optima). However, in their classic textbook on convex optimization Boyd and Vandenberghe (2004) study problems with explicit functions, whereas simulation problems have implicit functions that are determined by the underlying simulation model. In this paper, we use a metamodel to approximate such an implicit function (also see Nesterov 2003, pp. 171-172).Metamodels (also called response surfaces, emulators, etc.) serve sensitivity analysis of the simulation models and optimization of the simulated systems. There are several types of metamodels, but the most popular types are linear regression analysis and Kriging (or Gaussian process) models; many references to various types of metamodels are given by Kleijnen 2008, p. 8. Well-known types of monotonic regression models are isotonic regression and "rank" regression; see Kleijnen 2008, pp. 98, 162. We, however, focus on Kriging. Monotonic Kriging metamodels are also examined by Kleijnen and van Beers (2011); we summarize and update that publication, and extend it to convexity.To estimate the Kriging metamodel, we simulate (say) n combinations (or points) x i of the k ≥ 1 simulation inputs; we replicate these combinations m i times (i = 1, ..., n) We assume that the simulation model is expensive; i.e., the simulation requires much computer time to obtain the outputs w i;r (r = 1, ..., m i ), so the set of input/output (I/O) data may be so small that "classic" Kriging does not preserve the assumed characteristic, and shows wiggling (erratic) behavior. We therefore derive bootstrapp...

show abstract

“…Contemporary experimentation research is making theoretical contributions in both categories. For example, within the area of search experimentation, the formal techniques of meta-modelling (MM) (eg Kleijnen and Deflandre, 2006;Kleijnen, 2009;Ankenman et al, 2010;Poropudas and Virtanen, 2011;Chen et al, 2013;Kleijnen and van Beers, 2013), and simulation optimisation (SO) (eg Fu, 2002;Hong and Nelson, 2006;Subramaniam and Gosavi, 2007;Bettonvil et al, 2009;Kleijnen et al, 2010;Halim and Seck, 2011;Chang, 2012;Xu et al, 2013) and design of experiments (DoE) (eg Kleijnen, 2005;Sanchez et al, 2012;Ajdari and Mahlooji, 2014) are all active areas of research. The value of this research is arguably only fully realised once it has been transferred into common practice and applied to real-world problems (Taylor et al, 2009).…”

Section: Introductionmentioning

confidence: 99%

The use of search experimentation in discrete-event simulation practice

Hoad

Monks

O’Brien

2015

Journal of the Operational Research Society

View full text Add to dashboard Cite

The practice of DES experimentation has not been rigorously assessed in over a decade. Past studies of DES practice report little transfer of experimentation theory into real-world application. We conducted an international survey of over 300 modellers to investigate the extent to which simulation optimisation, meta-modelling and design of experiments are used in practice. Over the last decade there has been substantial growth in the use of optimisation and to a lesser extent design of experiments to tackle practical problems. However, users rarely make use of optimisers bundled with commercial software, opting instead for custom or third-party solutions. Outside of academia, the use of methods is hampered by a lack of application knowledge and a persisting view that such techniques are not necessary. It is clear that academics must not become complacent regarding the dissemination of theory into common practice and continue to reach out to industry users.

show abstract

Monotonicity-Preserving Bootstrapped Kriging Metamodels for Expensive Simulations

Cited by 20 publications

References 32 publications

A Bayesian metamodeling approach for stochastic simulations

A Bayesian metamodeling approach for stochastic simulations

Convex and monotonic bootstrapped Kriging

The use of search experimentation in discrete-event simulation practice

Contact Info

Product

Resources

About