Taming the Complexity of Artifact Reproducibility

Flittner, Matthias; Bauer, Robert; Rizk, Amr; Geissler, Stefan; Zinner, Thomas; Zitterbart, Martina

doi:10.1145/3097766.3097770

Cited by 4 publications

(4 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, according to [69], the "publish or perish" mentality is a significant problem: "Innovative findings produce the rewards of publication, employment and tenure; replicated findings produce a shrug." [68] and [70] suggest that in the future reproducible submissions should always be the default and that doing reproducible research will become imperative [6]. To that end, scientists, institutions and funding agencies have been pushing for the development of methodologies and tools that preserve software artifacts.…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Reproducibility in Computing Research: An Empirical Study

Raghupathi

Ren

2022

IEEE Access

View full text Add to dashboard Cite

In computing, research findings are often anecdotally faulted for not being reproducible. Numerous empirical studies have analyzed the reproducibility of a variety of research. Our objective, in this study, is to quantify the current state of reproducibility of research in computing based on prior research, using three reproducibility factors-Method, Data and Experiment-to measure three different degrees of reproducibility. Twenty-five variables traditionally utilized to document reproducibility are identified and grouped into three factors, namely Method, Data and Experiment. These variables describe the extent to which these factors are documented for each paper. Approximately 100 randomly selected research papers from the International Conference on Information Systems series, for the year 2019, are surveyed. Our findings suggest that none of the papers documented all the variables. In fact, the results show that relatively few variables for each factor are documented. Some of the variables vary across different categories of papers, and most papers fail in at least one of the factors. Reproducibility scores decrease with increased documentation requirements. Reproducibility may improve over time, as researchers prioritize reproducibility and utilize methods that ensure reproducibility. Research documentation in computing is remarkably limited, resulting in a dearth of reproducible factors. Future research may study the shifts and trends in reproducibility over time. Meanwhile, researchers and publishers must increase their focus on the reproducibility aspects of their papers. This study contributes to our understanding of the status quo of reproducibility in computing research.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

“…Certainly, reproducibility should automatically be a critical consideration of every research paper [4]. Not only does reproducibility allow researchers to build on published results but it also facilitates the review process [5]- [6]. Reproducible research is becoming an imperative, ensuring transparency and building trust.…”

Section: Introductionmentioning

confidence: 99%

Reproducibility in Computing Research: An Empirical Study

Raghupathi

Ren

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…A number of proposals have been made to improve artifacts: Collberg and Proebsting (2016) propose that authors include "sharing contracts" with their papers that give basic information about the artifacts for that paper, including the length of time and extent (e.g., bug fixes, feature requests) to which those artifacts will be supported. In the field of Software Defined Networks, Flittner et al (2017) propose that artifacts should be accompanied by meta-artifacts, which describe the tools and parameters that were used during an evaluation. Stodden (2009b,a) proposes the Reproducible Research Standard, a licensing framework that promotes the sharing of artifacts and ensures that authors are credited for their work.…”

Section: Related Workmentioning

confidence: 99%

“…To minimise such issues, creators should improve communication by clearly documenting (e.g., via a README) the important details of their artifacts. Below, we take inspiration from the ideas of sharing contracts (Collberg and Proebsting, 2016), meta-artifacts (Flittner et al, 2017), and data sharing agreements (Basili et al, 2007), discussed in further detail in Section 7, to identify some of those important details.…”

Section: Creatorsmentioning

confidence: 99%

Understanding and Improving Artifact Sharing in Software Engineering Research

Timperley,

Herckis,

Goues

et al. 2020

Preprint

View full text Add to dashboard Cite

In recent years, many software engineering researchers have begun to include artifacts alongside their research papers. Ideally, artifacts, which include tools, benchmarks, data, and more, support the dissemination of ideas, provide evidence for research claims, and serve as a starting point for future research. This often takes the form of a link in the paper pointing to a website containing these additional materials. However, in practice, artifacts suffer from a variety of issues that prevent them from fully realising that potential.To help the software engineering community realise the potential of artifacts, we seek to understand the challenges involved in the creation, sharing, and use of artifacts. To that end, we perform a mixed-methods study including a publication analysis and online survey of 153 software engineering researchers. We apply the established theory of diffusion of innovation, and draw from the field of implementation science, to make evidence-based recommendations.By analysing the perspectives of artifact creators, users, and reviewers, we identify several high-level challenges that affect the quality of artifacts including mismatched expectations between these groups, and a lack of sufficient reward for both creators and reviewers. Using diffusion of innovation as a framework, we analyse how these challenges relate to one another, and build an understanding of the factors that affect the sharing and success of artifacts. Finally, using principles from implementation science, we make evidence-based recommendations for specific sub-communities (e.g., students and postdocs, artifact evaluation committees, funding bodies, and professional organisations) to improve the quality of artifacts.

show abstract