A Survey on Artifacts from CoNEXT, ICN, IMC, and SIGCOMM Conferences in 2017

Abstract

Reproducibility of artifacts is a cornerstone of most scientific publications. To improve the current state and strengthen ongoing community efforts towards reproducibility by design, we conducted a survey among the papers published at leading ACM computer networking conferences in 2017: CoNEXT, ICN, IMC, and SIGCOMM. The objective of this paper is to assess the current state of artifact availability and reproducibility based on a survey. We hope that it will serve as a starting point for further discussions to encourage researchers to ease the reproduction of scientific work published within the SIGCOMM community. Furthermore, we hope this work will inspire program chairs of future conferences to emphasize reproducibility within the ACM SIGCOMM community as well as will strengthen awareness of researchers.

[Tool] Designing Replicable Networking Experiments With Triscale

“…Variability in experiment design and data analysis [6,59,65] Other factors hindering replicability Lack of documentation [6,44,76] Artifacts unavailability/unusability [23,29] Uncontrollability of the exp. conditions [18,53,57,86,87] Variability of hardware and software behavior [12,50,75] problem which stems from several factors (see Table 1).…”

“…To address this "replicability crisis" [7], many efforts aiming to incentivize a rigorous experimentation have gained momentum in computer science, including e.g., ACM's badging system for publications [1]. In the networking community, especially challenged by the need to carry out experiments in dynamic and uncontrollable conditions [18,53], several workshops [5,16,32], surveys [29], and guidelines [6,44,54,65] have raised awareness on the replicability problem and promoted better experimentation practices. This large body of work mostly offers qualitative statements on how an experiment should be performed and documented.…”

The Dagstuhl beginners guide to reproducibility for experimental networking research

“…The Stanford University Reproducibility course [26] is a good example of how students can take published research and attempt to reproduce and document the findings. A list of accepted papers in SIGCOMM-sponsored conferences that released artifacts were recently (2017) surveyed and a compiled list has been made available [15]. The recently established SIG-COMM Artifacts Evaluation Committee carried this initiative forward and applied badges to accepted papers in SIGCOMMsponsored events in 2018.…”

Making simulation results reproducible—Survey, guidelines, and examples based on Gradle and Docker

“…The open question part of the survey indicated a need for encouragement for publishing reproducible work or for papers that attempt to reproduce or refute earlier results. Flittner et al (2018) conducted an analysis of papers from four different ACM conferences held in 2017. This study found that the type of artifacts can differ significantly between different communities.…”