Automating the runtime performance evaluation of simulation algorithms

Abstract

Simulation algorithm implementations are usually evaluated by experimental performance analysis. To conduct such studies is a challenging and time-consuming task, as various impact factors have to be controlled and the resulting algorithm performance needs to be analyzed. This problem is aggravated when it comes to comparing many alternative implementations for a multitude of benchmark model setups. We present an architecture that supports the automated execution of performance evaluation experiments on several levels. Desirable benchmark model properties are motivated, and the quasi-steady state property of such models is exploited for simulation end time calibration, a simple technique to save computational effort in simulator performance comparisons. The overall mechanism is quite flexible and can be easily adapted to the various requirements that different kinds of performance studies impose. It is able to speed up performance experiments significantly, which is shown by a simple performance study.

Exploring the performance of spatial stochastic simulation algorithms

“…We studied more than 150 model setups by combining several JAMES II components to semiautomatically analyze algorithm performance: a performance database, a replication technique that allows to focus on the best-performing simulation setups, and a mechanism that selects a suitable simulation end time for a given model setup [34]. Executions that exceeded a predetermined timespan were aborted.…”

“…We therefore try to predict which algorithm is faster by applying the J48 decision tree learner from the WEKA machine learning toolkit [36, p. 159 et sqq.]. The learner is provided with features of the investigated model Simulation times vary across model set-ups, as their complexity differs strongly (they have been calibrated automatically [34] to let NSM run for %300 s, model setups with d = 1, P = 100 where so easy that even simulating until time 100 did not impose greater load for NSM). Unfinished executions were aborted after 10 min.…”

SafeBTW: A Scalable Optimistic Yet Non-risky Synchronization Algorithm

“…In the first stage, newly generated event-scheduling messages are allowed to be sent out, causing more events to be processed with risk, whereas in the second stage eventscheduling messages are not allowed to be sent out until they are determined to be legitimate. But under our pure performance test using PHOLD model [3], BTW algorithm also experienced performance degradation as the number of participating nodes increased. After careful studying, we identified that the main cause for this degradation is excessive overhead introduced by cascading rollback operations, which coincides with Steinman [2], Lubachevsky [4] and Turner's [5] research.…”

Using AI Planning to Automate the Performance Analysis of Simulators

“…Here, a 'quasi-steady state' means that performance-relevant model features (e.g., model structure, number of entities, etc.) are relatively stable during a simulation run, so that execution time is approximately proportional to the number of executed simulation events (see [6]).…”

“…To make a valid comparison, however, one also needs to know for how many simulation events the given model should be simulated until execution times are not dominated by stochastic noise or warm-up cost anymore. This data can be provided, for example, by a 'calibration' action implementing an approach as described in [6], and which is thus declared here as well. If there would be no such action (or any alternative), the user would be notified that no plan could be found.…”