Using Ranking and Selection to “Clean Up” after Simulation Optimization

Abstract

In this paper we address the problem of finding the simulated system with the best (maximum or minimum) expected performance when the number of systems is large and initial samples from each system have already been taken. This problem may be encountered when a heuristic search procedure-perhaps one originally designed for use in a deterministic environment-has been applied in a simulationoptimization context. Because of stochastic variation, the system with the best sample mean at the end of the search procedure may not coincide with the true best system encountered during the search. This paper develops statistical procedures that return the best system encountered by the search (or one near the best) with a prespecified probability. We approach this problem using combinations of statistical subset selection and indifference-zone ranking procedures. The subset-selection procedures, which use only the data already collected, screen out the obviously inferior systems, while the indifference-zone procedures, which require additional simulation effort, distinguish the best from the less obviously inferior systems.

Comparison of Kriging-Based Methods for Simulation Optimization with Heterogeneous Noise

“…One could include, for instance, a R&S procedure that acts as a "clean-up" in the identification step (see, e.g., the procedure proposed in Boesel, Nelson, & Kim, 2003 ); the aim of this R&S procedure is to discard the bad alternatives in T and find a sufficiently good one using as few replications as possible. Alternatively, R&S could be used in the optional replication step (see Fig.…”

Simulation optimization: A comprehensive review on theory and applications

“…Nelson et al (2001) present a general theory and procedures for reducing sampling efforts in a two-stage indifference-zone approach when the number of alternatives is large. Boesel (2000) and Boesel et al (2003) study the problem of finding the best system when the number of systems is large and initial samples from each system have already been taken. These articles develop statistical procedures that identify the best system encountered in the search by using a variety of approaches including subset selection and indifference-zone.…”

Fully sequential selection procedures with a parabolic boundary

“…If there exist systems whose means are within δ of the best, then the experimenter is indifferent to which of these is selected. Recent references include Nelson et al (2001), Boesel et al (2003), and Nelson (2001, 2005).…”