Opportunity Cost and OCBA Selection Procedures in Ordinal Optimization for a Fixed Number of Alternative Systems

Abstract

Ordinal optimization offers an efficient approach for simulation optimization by focusing on ranking and selecting a finite set of good alternatives. Because simulation replications only give estimates of the performance of each alternative, there is a potential for incorrect selection. Two measures of selection quality are the alignment probability or the probability of correct selection (P{CS}), and the expected opportunity cost E[OC], of a potentially incorrect selection. Traditional ordinal optimization approaches focus on the former case. This paper extends Chen's optimal computing budget allocation (OCBA) approach, which allocated replications to improve P{CS}, to provide the first OCBA-like procedure that optimizes E[OC] in some sense. The procedure performs efficiently in numerical experiments.

Paradoxes in Learning and the Marginal Value of Information

“…Indeed, existing general purpose allocation techniques for this single-stage R&S problem with linear loss, LL(B) (Chick and Inoue 2001) and OCBA for linear loss (He et al 2007), both recommend the equal allocation in the homogeneous case. (Although they recommend equal allocation in the homogeneous case, they make other recommendations in other cases).…”

“…For example, Figure 5(b) shows that the best proportion when N = 25 is 0 36, which is quite far from 0 84. The predominant methods for finding good single-stage allocations, LL(B) (Chick and Inoue 2001) and the OCBA (Chen 1996, He et al 2007), approximate this optimal asymptotic proportion and then suggest that we allocate our available (finite) sampling budget according to this proportion. The difference between asymptotic and finite-horizon optimal proportions and its effect on these policies may be an interesting topic for future research.…”

The Knowledge Gradient for Optimal Learning

“…Chick and Inoue [20] introduces the LL(B) strategy, which maximizes the linear loss with measurement budget B. He and Chick [21] introduce an OCBA procedure for optimizing the expected value of a chosen design, using the Bonferroni inequality to approximate the objective function for a single stage. A common strategy in simulation is to test different parameters using the same set of random numbers to reduce the variance of the comparisons.…”

The knowledge-gradient stopping rule for ranking and selection

“…Within this body of work, a number of staged and fully sequential Bayesian ranking and selection techniques have been recently proposed including Chen, Dai, and Chen (1996), Chen, Lin, Yücesan, and Chick (2000), Chick and Inoue (2001b), Chick and Inoue (2001a), Chen, He, and Fu (2006), He, Chick, and Chen (2007). These techniques optimize average-case instead of worst-case performance, and so are generally less conservative than techniques using the indifference zone formulation.…”

“…The rule adopts the linear loss function, which penalizes according to the difference in value between the chosen option and the best, contrasting it with another common choice, 0 − 1 loss, which penalizes a constant 1 for failing to find the best alternative. Other algorithms designed for the linear loss objective function under independent normal samples include LL(S) (Chick and Inoue 2001b) and OCBA for linear loss (He, Chick, and Chen 2007).…”