CRISIS, WHAT CRISIS – DOES REPRODUCIBILITY IN MODELING & SIMULATION REALLY MATTER?

Abstract

How important it is to our discipline that we can reproduce the results of Modeling & Simulation (M&S) research? How important is it to be able to (re)use the models, data, and methods described in simulation publications to reproduce published results? Is it really that important or are the lessons and experiences described in a paper enough for us to build on the work of others? At the 2016 Winter Simulation Conference, a panel considered opinions on reproducibility in discrete-event simulation. This article builds on these and asks if there really is a reproducibility crisis in M&S? A diverse range of views on the subject are presented including reflections on the reproducibility in terms of the art and science of simulation, the frustrations of poor reproducibility, perspectives from the industrial production & logistics community, the wider context of open science and artefact sharing, and the role of provenance beyond reproducibility. INTRODUCTIONWhat does reproducibility mean in terms of Modeling & Simulation? Does reproducibility matter? To what degree is reproducibility possible? Could reproducibility policies have a negative impact on publishing in M&S? These questions were amongst some initially posed to our panelists as a starting point to gather a range of views on this potentially important topic. This panel paper presents a range of views on the subject and contributes to the on-going debate about the role of reproducibility in our discipline.

Revisiting the richness of integrated vehicle and crew scheduling

“…Differently, this topic relates to replicating studies performed on developed models in mathematical programming, discrete event simulation, etc. (see, e.g., the discussion in Taylor et al 2018). It seems still seldom that a repository is set up with the underlying data and models as well as algorithms being provided for readers.…”

Network Cards: concise, readable summaries of network data

“…Alongside the literature's exponential growth, many fields of research are in the midst of a replication crisis, where past work has been called into question [14,15,16,17]. Causes of this crisis include poor statistical practice [18,19,20] and poor data documentation [21,22,23]. Documenting the provenance of data is crucial for data-driven studies of networks, and there is a real need for a systematic, standard way to describe the various details of a network dataset, details that are not strictly part of the network topology itself and so are often lost as researchers share data files describing that topology but nothing else.…”

A Model-Driven Approach for Conducting Simulation Experiments

“…Based on the computed field distributions, the biological response of the stimulated biological sample can be linked to certain specifications of the electrical stimulation set-up. A global SA 6 is used to evaluate the influence of the dielectric parameters on the electric field strength at specific locations of the cells.…”

“…To support simulation experiments, a clear separation of concerns between model and experiment is necessary [5]. Following the reproducibility crisis [6], a variety of approaches have been developed for capturing simulation experiments explicitly and machine-accessibly. This is reflected, for example, in domain-specific languages such as SESSL ("Simulation Experiment Specification via a Scala Layer") [7], [8] and SED-ML ("Simulation Experiment Description Markup Language"), or reporting guidelines such as MIASE ("Minimum Information About a Simulation Experiment") [9] and the reporting guidelines for finite element analysis studies in biomechanics [10].…”