Sensitivity of computer simulation experiments to errors in input data

Abstract

This paper compares two methods of assessing variability in simulation output. The methods make specific allowance for two sources of variation: that caused by uncertainty in estimating unknown input parameters (parameter uncertainty), and that caused by the inclusion of random variation within the simulation model itself (simulation uncertainty). The first method is based on classical statistical differential analysis; we show explicitly that, under general conditions, the two sources contribute separately to the total variation.In the classical approach, certain sensitivity coefficients have to be estimated. The effort needed to do this becomes progressively more expensive, increasing linearly with the number of unknown parameters. Moreover there is an additional difficulty of detecting spurious variation when the number of parameters is large. It is shown that a parametric form of bootstrap sampling provides an alternative method which does not suffer from either problem.For illustration, simulation of the operation of a small (4-node) computer communication network is used to compare the accuracy of estimates using the two methods. A larger, more realistic, (30-node) network is presented showing how the bootstrap method becomes competitive when the number of unknown parameters is large.

Reliable simulation with input uncertainties using an interval-based approach

“…The first term is the simulation variance, and the second term is the input parameter variance. The early work of Cheng and Holland (Cheng 1994, Cheng and Holland 1997, Cheng and Holland 1998 did not include the bias in the mean square error (MSE) of the parameters. The failure of including the bias is substantial in the sense that the simulation output confidence intervals are conservative, hence, the variance is overestimated (Henderson 2003).…”

“…Cheng and Holland (Cheng 1994, Cheng and Holland 1997, Cheng and Holland 1998) developed a delta method for input uncertainties. The framework was also adopted by Wilson (2001a, 2001b).…”

The role of learning on industrial simulation design and analysis

“…This practice of simulation design and analysis leads to not only inconsistent estimates for the mean performance measures but also inconsistent coverage of the confidence intervals (Barton 2012). A close look at the stochastic simulation literature on accounting for parameter uncertainty shows that Cheng and Holland (1997) are the first to show the dependence of the simulation output on stochastic and parameter uncertainties using the delta and parametric bootstrap methods. These solution approaches are followed by the two-point method in Cheng and Holland (1998) as well as an alternative approach that particularly considers the simulation bias in Cheng and Holland (2004).…”

A kernel approach to estimating the density of a conditional expectation

“…Many of these methods impose a probability distribution on the input distributions and parameters. For example, this is the case in the papers Cheng (1994), Cheng and Holland (1997), Cheng and Holland (1998), Cheng and Holland (2003), Chick (2001), Zouaoui and Wilson (2001b), Zouaoui and Wilson (2001a). See Henderson (2003) for further discussion.…”