Program slice distribution functions

Abstract

Unexpected behaviors in simulations require explanation, so that decision makers and subject matter experts can separate valid behaviors from design or coding errors. Validation of unexpected behaviors requires accumulation of insight into the behavior and the conditions under which it arises. Stochastic simulations are known for unexpected behaviors that can be difficult to recreate and explain. To facilitate exploration, analysis and understanding of unexpected behaviors in stochastic simulations we have developed a novel approach, called Program Slice Distribution Functions (PSDFs), for quantifying the uncertainty of the dynamic program slices (simulation executions) causing unexpected behaviors. Our use of PSDFs is the first approach to quantifying the uncertainty in program slices for stochastic simulations and extends the state of the art in analysis and informed decision making based on simulation outcomes. We apply PSDFs to a published epidemic simulation and describe how users can apply PSDFs to their own stochastic simulations. INTRODUCTIONExploratory simulations have entered the mainstream of critical public policy and research decision-making practices (Cha 2005, Whipple 1996, Hooke 2000, Elderd 2006, National Science Foundation 2006, Arthur 1999. Public policy-makers and scientists look to these simulations for insight, trends and likely outcomes. Unfortunately, methods for gaining insight into unexpected outcomes with uncertain validity, as related to model design and simulation implementation and use, have not kept pace. We present a new approach to automating and quantifying some of the insight-gathering process for identifying, analyzing and understanding sources of unexpected behaviors in exploratory stochastic simulations. The specifications of exploratory simulations are often incomplete because the application domain is poorly understood; the purpose of the simulation is to explore the domain of interest (Trenouth 1991). Writing the simulation becomes a theory construction task where the software is the expression of the theory (Wielinga 1978). Trusted simulations in the same application domain, datasets from physical experiments, and subject matter expert opinions are used to test the theory. This is what gives exploratory simulations their experimental nature. Those behaviors that are not defined in the specification and do not match the behavior of other trusted simulations, data sets from physical experiments or subject matter expert opinions are unexpected behaviors. Unexpected behaviors require understanding and explanation to determine if the behavior is an error or new knowledge in the application domain.The daunting nature of quantifying, analyzing and understanding uncertainty in model design and simulation outcomes is evident in the results of epidemiology studies conducted this century. Epidemiologists have addressed the question of government level actions and reactions regarding the spread of infectious diseases such as smallpox and bird flu. Should a comprehensive vacc...