Work smarter, not harder: A tutorial on designing and conducting simulation experiments

Abstract

Simulation models are integral to modern scientific research, national defense, industry and manufacturing, and in public policy debates. These models tend to be extremely complex, often with thousands of factors and many sources of uncertainty. To understand the impact of these factors and their interactions on model outcomes requires efficient, high-dimensional design of experiments. Unfortunately, all to often, many large-scale simulation models continue to be explored in ad hoc ways. This suggests that more simulation researchers and practitioners need to be aware of the power of experimental design in order to get the most from their simulation studies. In this tutorial, we demonstrate the basic concepts important for design and conducting simulation experiments, and provide references to other resources for those wishing to learn more. This tutorial (an update of previous WSC tutorials) will prepare you to make your next simulation study a simulation experiment. INTRODUCTIONIn June 2008, a new supercomputer called the "Roadrunner" was unveiled. This bank of machines was assembled from components originally designed for the video game industry; it costs $133 milion, and is capable of doing a petaflop (a quadrillion operations per second). The New York Times coverage stated that "petaflop machines like Roadrunner have the potential to fundamentally alter science and engineering" by allowing researchers to "ask questions and receive answers virtually interactively" and "perform experiments that would previously have been impractical" (Markoff 2008). Four years later, IBM's "Sequoia" supercomputer is the new world leader, with 16 petaflop capability. Yet let's take a closer look at the practicality of a brute-force approach to simulation experiments. Suppose a simulation has 100 factors, each factor has two levels (say, low and high) of interest, and we decide to look at each combination of these 100 factors. A single replication of this experiment for simulation that runs as fast as a single operation would take over 2.5 million years on the Sequoia and over 40 million years on the Roadrunner! Efficient design of experiments can break this curse of dimensionality at a tiny fraction of the cost. For example, suppose we want study 100 factors and all their two-way interactions. One screening design we could use (a resolution 5 fractional factorial, described in Section 3.3) specifies 32768 specific combinations of the factor levels to evaluate. How quickly can we finish such an experiment? On a desktop computer with a simulation that takes a full second to run, each replication of this experiment takes under 9.5 hours; even if the simulation takes a more reasonable one minute to run, we can finish this experiment on an 8-core desktop (under $3,000) in 2.85 days. Other designs are even more efficient, and provide more detailed insights into the simulation model's behavior.The field called Design of Experiments (DOE) has been around for a long time. Many of the classic experimental designs can be used in ...

Simulation data management for adaptive design of experiments: A litterature review

“…[1] for additional details on DoE properties). Thus, selection of DoE method may be a very time-consuming process [44] developed a design comparison chart to help designers in their choices, but it is non-exhaustive. Thus, there still exists a need for classification and comparison of DoE methods to support designers' decisions.…”

The exponential expansion of simulation in research

“…Furthermore, the concept of "optimality" permeates the practical real world where "sampling is expensive-the goal is to take no more samples than absolutely necessary." That is, even if you could spare the time to run as many replications of your complex model as you wanted, you do not have to (Sanchez and Wan 2012). The optimality of efficient design is holding hands with the impressive capabilities of modern computers to create simulation models that represent a powerful force that is anything but "brute."…”

“…Of course, another advantage of many space-filling designs, such as NOLHs, is that they are robust in the sense that they allow analysts to explore a breadth of diverse possible metamodels (Kleijnen et al 2005;Vieira Jr. et al 2011). For more on the use of experimental design for simulation (including additional references), please see the tutorial by Sanchez and Wan (2012) earlier in this Proceedings. Advances in simulation optimization (Fu 2002) are also providing new opportunities for simulation analysts; there are several sessions in the 2012 WSC Proceedings, and an entire track in the 2011 WSC Proceedings.…”

A model-driven engineering approach to simulation experiment design and execution

“…DoE has provided scientists with a means of reproducing and replicating physical experiments for many years. By making some adaptations to its methodology, we can use it for the same purpose in modeling and simulation (Sanchez and Wan 2012;Lorscheid, Heine, and Meyer 2012;Kelton 2000;Kleijnen 2005). In our previous work, we have taken a similar position (Ledet et al 2014) and have proposed using, in addition to DoE, a Model-Driven Engineering (MDE) framework for managing simulation experiments throughout their entire life-cycle ).…”

“…In this paper, we discuss some advances we achieved in implementing such a system. The majority of the work found in the literature focuses on the proper techniques required for designing simulation experiments (Kleijnen 2005;Kelton 2000;Lorscheid, Heine, and Meyer 2012;Sanchez and Wan 2012). Though there are emerging tools for managing experiments (Ioannidis et al 1997;Perrone, Main, and Ward 2012;Leye and Uhrmacher 2012), they do not encompass the entire experimental life-cycle, which includes adaptation of experiments as learning takes place.…”