Designs for Large‐Scale Simulation Experiments, with Applications to Defense and Homeland Security

Sanchez, Susan M.; Lucas, Thomas W.; Sánchez, Paul; Nannini, Christopher J.; Wan, Hong

doi:10.1002/9781118147634.ch12

Cited by 28 publications

(21 citation statements)

References 25 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yet even using the fastest computers in large computing clusters cannot overcome the ''curse of dimensionality,'' so a brute-force approach will not work when the number of factors is large. These all suggest that having a framework to generate customized designs with limited number of runs is very useful as part of a portfolio of designs that facilitate large-scale simulation experiments (see Sanchez et al (2010) for further discussion and applications). Our ongoing research focuses on the creation of larger orthogonal designs (more than 100 factors); on the study of the imbalance effect on the achievement of design orthogonality; and on the incorporation of qualitative factors into the designs.…”

Section: Discussionmentioning

confidence: 99%

Generating and improving orthogonal designs by using mixed integer programming

Vieira

Sanchez

Kienitz

et al. 2011

European Journal of Operational Research

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

Generating and improving orthogonal designs by using mixed integer programming

Vieira

Sanchez

Kienitz

et al. 2011

European Journal of Operational Research

View full text Add to dashboard Cite

“…Those creating scenarios can use screening experiments early in the model-building process to aid in model verification and help set suitable factor ranges. The international consortium looking at the model-based ship design project will benefit from having information about the robustness of the findings-we advocate that it is better to experiment on a large number of factors and seek a simplified model than to limit the factors from the outset (Sanchez et al 2012). Finally, by examining simulation models that are being used for decision-making, we will continue to identify challenges that would benefit from further methodological research.…”

Section: Simulation Experimentsmentioning

confidence: 99%

Simulation screening experiments using Lasso-optimal supersaturated design and analysis: A maritime operations application

Xing¹,

Wan²,

Zhu³

et al. 2013

2013 Winter Simulations Conference (WSC)

Self Cite

View full text Add to dashboard Cite

Screening methods are beneficial for studies involving simulations that have a large number of variables where a relatively small (but unknown) subset is important. In this paper, we show how a newly proposed Lasso-optimal screening design and analysis method can be useful for efficiently conducting simulation screening experiments. Our approach uses new criteria for generating supersaturated designs, and a new algorithm for selecting the optimal tuning parameters for Lasso model selection. We generate a 24x69 Lasso optimal supersaturated design, illustrate its potential with a numerical evaluation, and apply it to an agent-based simulation of maritime escort operations in the Strait of Gibraltar. This application is part of a larger project that seeks to leverage simulation models during the ship design process, and so construct ships that are both cost effective and operationally effective. The supersaturated screening design has already proved beneficial for model verification and validation.

show abstract

“…For more on the philosophy and tactics of designing simulation experiments, examples of graphical methods that facilitate gaining insight into the simulation model's performance, and extensive literature surveys, we refer the reader to Sanchez et al (2012) or Kleijnen et al (2005). Books that discuss experimental designs for simulation include Santner, Williams, and Notz (2003), Law (2007), and Kleijnen (2007).…”

Section: Finding Out Morementioning

confidence: 99%

“…Although the examples in this paper are very simple simulation models, the same types of designs have been extremely useful for investigating more complex simulation models in a variety of application areas. For a detailed discussion of the philosophy and tactics of simulation experiments, a more extensive catalog of potential designs, and a comprehensive list of references, see Kleijnen et al (2005) or Sanchez et al (2012); other useful references are Kleijnen (2007), Chapter 12 of Law (2007), or Sanchez (2009).…”

Section: Introductionmentioning

confidence: 99%

Work smarter, not harder: A tutorial on designing and conducting simulation experiments

Sanchez

Wan

2012

Proceedings Title: Proceedings of the 2012 Winter Simulation Conference (WSC)

Self Cite

View full text Add to dashboard Cite

Simulation models are integral to modern scientific research, national defense, industry and manufacturing, and in public policy debates. These models tend to be extremely complex, often with thousands of factors and many sources of uncertainty. To understand the impact of these factors and their interactions on model outcomes requires efficient, high-dimensional design of experiments. Unfortunately, all to often, many large-scale simulation models continue to be explored in ad hoc ways. This suggests that more simulation researchers and practitioners need to be aware of the power of experimental design in order to get the most from their simulation studies. In this tutorial, we demonstrate the basic concepts important for design and conducting simulation experiments, and provide references to other resources for those wishing to learn more. This tutorial (an update of previous WSC tutorials) will prepare you to make your next simulation study a simulation experiment. INTRODUCTIONIn June 2008, a new supercomputer called the "Roadrunner" was unveiled. This bank of machines was assembled from components originally designed for the video game industry; it costs $133 milion, and is capable of doing a petaflop (a quadrillion operations per second). The New York Times coverage stated that "petaflop machines like Roadrunner have the potential to fundamentally alter science and engineering" by allowing researchers to "ask questions and receive answers virtually interactively" and "perform experiments that would previously have been impractical" (Markoff 2008). Four years later, IBM's "Sequoia" supercomputer is the new world leader, with 16 petaflop capability. Yet let's take a closer look at the practicality of a brute-force approach to simulation experiments. Suppose a simulation has 100 factors, each factor has two levels (say, low and high) of interest, and we decide to look at each combination of these 100 factors. A single replication of this experiment for simulation that runs as fast as a single operation would take over 2.5 million years on the Sequoia and over 40 million years on the Roadrunner! Efficient design of experiments can break this curse of dimensionality at a tiny fraction of the cost. For example, suppose we want study 100 factors and all their two-way interactions. One screening design we could use (a resolution 5 fractional factorial, described in Section 3.3) specifies 32768 specific combinations of the factor levels to evaluate. How quickly can we finish such an experiment? On a desktop computer with a simulation that takes a full second to run, each replication of this experiment takes under 9.5 hours; even if the simulation takes a more reasonable one minute to run, we can finish this experiment on an 8-core desktop (under $3,000) in 2.85 days. Other designs are even more efficient, and provide more detailed insights into the simulation model's behavior.The field called Design of Experiments (DOE) has been around for a long time. Many of the classic experimental designs can be used in ...

show abstract

Designs for Large‐Scale Simulation Experiments, with Applications to Defense and Homeland Security

Cited by 28 publications

References 25 publications

Generating and improving orthogonal designs by using mixed integer programming

Generating and improving orthogonal designs by using mixed integer programming

Simulation screening experiments using Lasso-optimal supersaturated design and analysis: A maritime operations application

Work smarter, not harder: A tutorial on designing and conducting simulation experiments

Contact Info

Product

Resources

About