On the Replicability of Experimental Tool Evaluations in Model-Based Development

Boll, Alexander; Kehrer, Timo

doi:10.1007/978-3-030-58167-1_9

Cited by 5 publications

(3 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The set by Chowdhury et al comprises a smaller Simulink model collection [34], a Stateflow model collection by the CoCo-Sim-Team [9], and many other projects from Matlab Central, Source-Forge, GitHub, and other sources such as web sites of Although critical open-source repository sites could have been missed, the set by Chowdhury et al covers a wide range of sources. Instead of using the provided dataset as it is, we re-collected a current snapshot in August 2020 7 consisting of all constituent Simulink models based on the information provided in the meta-data of the corpus of Chowdhury et al The main motivation for this new snapshot arises from several inconsistencies we found between the actual corpus and the results presented in [15]. According to personal correspondence with the authors, these inconsistencies may originate from only a subset of the entire corpus models being used in their study.…”

Section: Study Subjectsmentioning

confidence: 99%

“…Our corpus comprising all the projects and models used in this study, including the respective meta-data, is available in our replication package. 7 https://doi.org/10.6084/m9.figshare.13636589…”

Section: Study Subjectsmentioning

confidence: 99%

“…However, Simulink projects and models created and maintained in an industrial context are usually not pub-licly available due to confidentiality agreements or license restrictions [2,21,51,63]. Access to these models, in general, is limited, making research results hard if not impossible to replicate [7]. Publicly available projects do not reflect "real-world" models [6,12,13,31,33], which severely limits empirical research.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Characteristics, potentials, and limitations of open-source Simulink projects for empirical research

et al. 2021

Self Cite

View full text Add to dashboard Cite

Simulink is an example of a successful application of the paradigm of model-based development into industrial practice. Numerous companies create and maintain Simulink projects for modeling software-intensive embedded systems, aiming at early validation and automated code generation. However, Simulink projects are not as easily available as code-based ones, which profit from large publicly accessible open-source repositories, thus curbing empirical research. In this paper, we investigate a set of 1734 freely available Simulink models from 194 projects and analyze their suitability for empirical research. We analyze the projects considering (1) their development context, (2) their complexity in terms of size and organization within projects, and (3) their evolution over time. Our results show that there are both limitations and potentials for empirical research. On the one hand, some application domains dominate the development context, and there is a large number of models that can be considered toy examples of limited practical relevance. These often stem from an academic context, consist of only a few Simulink blocks, and are no longer (or have never been) under active development or maintenance. On the other hand, we found that a subset of the analyzed models is of considerable size and complexity. There are models comprising several thousands of blocks, some of them highly modularized by hierarchically organized Simulink subsystems. Likewise, some of the models expose an active maintenance span of several years, which indicates that they are used as primary development artifacts throughout a project’s lifecycle. According to a discussion of our results with a domain expert, many models can be considered mature enough for quality analysis purposes, and they expose characteristics that can be considered representative for industry-scale models. Thus, we are confident that a subset of the models is suitable for empirical research. More generally, using a publicly available model corpus or a dedicated subset enables researchers to replicate findings, publish subsequent studies, and use them for validation purposes. We publish our dataset for the sake of replicating our results and fostering future empirical research.

show abstract

Section: Study Subjectsmentioning

confidence: 99%

Section: Study Subjectsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Characteristics, potentials, and limitations of open-source Simulink projects for empirical research

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

Replicability of experimental tool evaluations in model-based software and systems engineering with MATLAB/Simulink

Boll

Vieregg

Kehrer

2022

Innovations Syst Softw Eng

View full text Add to dashboard Cite

Research on novel tools for model-based development differs from a mere engineering task by not only developing a new tool, but by providing some form of evidence that it is effective. This is typically achieved by experimental evaluations. Following principles of good scientific practice, both the tool and the models used in the experiments should be made available along with a paper, aiming at the replicability of experimental results. We investigate to which degree recent research reporting on novel methods, techniques, or algorithms supporting model-based development with MATLAB/Simulink meets the requirements for replicability of experimental results. Our results from studying 65 research papers obtained through a systematic literature search are rather unsatisfactory. In a nutshell, we found that only 31% of the tools and 22% of the models used as experimental subjects are accessible. Given that both artifacts are needed for a replication study, only 9% of the tool evaluations presented in the examined papers can be classified to be replicable in principle. We found none of the experimental results presented in these papers to be fully replicable, and 6% partially replicable. Given that tools are still being listed among the major obstacles of a more widespread adoption of model-based principles in practice, we see this as an alarming signal. While we are convinced that this situation can only be improved as a community effort, this paper is meant to serve as starting point for discussion, based on the lessons learnt from our study.

show abstract

Chorus Signal Positioning Algorithm based on Signal Acquisition Optimization with Mixing Equipment

Fan

2023

2023 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS)

View full text Add to dashboard Cite

On the Replicability of Experimental Tool Evaluations in Model-Based Development

Cited by 5 publications

References 41 publications

Characteristics, potentials, and limitations of open-source Simulink projects for empirical research

Characteristics, potentials, and limitations of open-source Simulink projects for empirical research

Replicability of experimental tool evaluations in model-based software and systems engineering with MATLAB/Simulink

Chorus Signal Positioning Algorithm based on Signal Acquisition Optimization with Mixing Equipment

Contact Info

Product

Resources

About