Toward a model-driven engineering framework for reproducible simulation experiment lifecycle management

Teran-Somohano, Alejandro; Dayibas, Orcun; Yılmaz, Levent; Smith, Alice E.

doi:10.1109/wsc.2014.7020116

Cited by 9 publications

(6 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been various studies on the reproducibility of programming issues (Fazzini et al, 2018;Horton and Parnin, 2018;Mondal et al, 2019;Moran et al, 2016;Mu et al, 2018;Rahman et al, 2020;Soltani et al, 2017;White et al, 2015;Yang et al, 2016), computational reproducibility (Dalle, 2012;Freire et al, 2012;Goecks et al, 2010;Liu and Salganik, 2019;Marwick, 2017), experimental reproducibility (Dit et al, 2015;Teran-Somohano et al, 2014), issues in reproducing research results (Boettiger, 2015;Cito and Gall, 2016;Grüning et al, 2018;Jimenez et al, 2017;de Oliveira Neto et al, 2015;Playford et al, 2016;Poldrack and Poline, 2015;Scheitle et al, 2017;Walters, 2013), and reproducibility issues in software engineering (Crick et al, 2014;Rodríguez-Pérez et al, 2018). However, to the best of our knowledge, we first investigate the reproducibility of issues reported in Stack Overflow questions in two popular programming languages -Java and Python.…”

Section: Related Workmentioning

confidence: 99%

The Reproducibility of Programming-Related Issues in Stack Overflow Questions

Mondal¹,

Rahman²,

Roy³

et al. 2021

Preprint

View full text Add to dashboard Cite

Software developers often look for solutions to their code-level problems using the Stack Overflow Q&A website. To receive help for coding problems, developers frequently submit questions that contain sample code segments along with the description of the programming issue. Unfortunately, it is not always possible to reproduce the issues they report from the code segments they provide. Issues that are not easily reproducible may impede questions from receiving prompt and appropriate solutions. We conducted an exploratory study on the reproducibility of issues discussed in 800 Stack Overflow questions (400 Java + 400 Python). We parsed, compiled, executed, and carefully examined the code segments from these questions in order to reproduce the reported programming issues, expending 300 person-hours of effort. The outcomes of our study are three-fold. First, we found that we can reproduce the issues reported by developers for approximately 68% of Java and 71% of Python code segments, whereas we were unable to reproduce the issues for approximately 22% of Java code segments and 19% of Python code segments. Of the issues that were reproducible, approximately 67% of the Java code segments and 20% of the Python code segments required minor or major modifications to reproduce the issues. Second, we also carefully investi-

show abstract

Section: Related Workmentioning

confidence: 99%

The Reproducibility of Programming-Related Issues in Stack Overflow Questions

Mondal¹,

Rahman²,

Roy³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Experimentation via simulation is not a new concept. Several recent works focus on managing simulation experiments throughout their entire life cycle [1][2][3]. With an eye toward reproducibility, these studies combine model-driven engineering concepts with intelligent software agents to aid in the design, batch execution, and iterative refinement of simulation experiments.…”

Section: Existing Support For Virtual Experimentationmentioning

confidence: 99%

Developing a vision for executing scientifically useful virtual biomedical experiments

Petersen

Hunt

2016

Agent-Directed Simulation Symposium (ADS 2016)

View full text Add to dashboard Cite

A virtual biomedical experiment (VBE) is a simulation of a wet-lab or clinical biomedical experiment. The goal of VBEs is to provide a scientifically useful method of in silico experimentation that can challenge concrete hypotheses and provide explanatory, mechanistic insight into the referent system. We envision virtual experimentation not as a supplement to traditional wet-lab experimentation, but rather as an essential part of the scientific method itself. The goal of this work is to lay preliminary groundwork for realizing this vision, through outlining requirements and describing agentbased models demonstrative of this vision. VBEs focus on reasoning by analogy; thus, a VBE includes model components analogous to particular relevant aspects of the referent experiment-from hypothesis formation to data analysis, and key concepts in between. We explore five exemplary categories of scientifically useful VBEs: hypothesis, experiment context, living counterparts, experiment agents, and measurements. We discuss how to develop model components of each category in the context of agent-based modeling. We highlight the importance of two overarching requirements: concreteness and modularity. Finally, we demonstrate this vision by describing an in silico liver model that partially fulfills the VBE vision and requirements, including components corresponding to each of the five VBE categories.

show abstract

“…Yet, specific instructions on how to perform each step of the simulation study for answering the research question are rarely given and the methodological appropriateness is often disregarded (Timm and Lorig 2015). To overcome these shortcomings the integrated assistance of the entire life-cycle of a simulation study is proposed, i.e., the design, execution, and evaluation of simulation experiments (Teran-Somohano et al 2014). For this purpose, existing approaches, scripts, services, and assistance functionalities for conducting simulation studies need to be logically linked with respect to existing procedure models.…”

Section: Introductionmentioning

confidence: 99%

Formal Specification of Hypotheses for Assisting Computer Simulation Studies

2017

TMS/DEVS Symposium on Theory of Modeling &Amp; Simulation (TMS/DEVS 2017)

View full text Add to dashboard Cite

The aim of computer simulation studies is to answer research questions by means of experiments. For providing reliable evidence, the procedure of the study needs to be aligned with the question and the steps of the study need to be adjusted and combined accordingly. Supporting this process is challenging as the identification, customization, and combination of adequate tools and techniques for the systematic design of simulation experiments with respect to a hypothesis is not trivial. Hence, for providing computer-aided assistance, a language for the specification of hypotheses with respect to the credible and reproducible testing of research questions in computer simulation is needed. In this paper, we propose an approach for formally specifying hypotheses that allows for automated hypothesis testing. Based on specified hypotheses, we demonstrate the assistance of simulation studies in terms of model parametrization and analysis of results with respect to the statistically sound evaluation of hypotheses.

show abstract

Toward a model-driven engineering framework for reproducible simulation experiment lifecycle management

Cited by 9 publications

References 16 publications

The Reproducibility of Programming-Related Issues in Stack Overflow Questions

The Reproducibility of Programming-Related Issues in Stack Overflow Questions

Developing a vision for executing scientifically useful virtual biomedical experiments

Formal Specification of Hypotheses for Assisting Computer Simulation Studies

Contact Info

Product

Resources

About