Publishing computational research - a review of infrastructures for reproducible and transparent scholarly communication

Abstract

Background: The trend toward open science increases the pressure on authors to provide access to the source code and data they used to compute the results reported in their scientific papers. Since sharing materials reproducibly is challenging, several projects have developed solutions to support the release of executable analyses alongside articles. Methods: We reviewed 11 applications that can assist researchers in adhering to reproducibility principles. The applications were found through a literature search and interactions with the reproducible research community. An application was included in our analysis if it (i) was actively maintained at the time the data for this paper was collected, (ii) supports the publication of executable code and data, (iii) is connected to the scholarly publication process. By investigating the software documentation and published articles, we compared the applications across 19 criteria, such as deployment options and features that support authors in creating and readers in studying executable papers. Results: From the 11 applications, eight allow publishers to self-host the system for free, whereas three provide paid services. Authors can submit an executable analysis using Jupyter Notebooks or R Markdown documents (10 applications support these formats). All approaches provide features to assist readers in studying the materials, e.g., one-click reproducible results or tools for manipulating the analysis parameters. Six applications allow for modifying materials after publication. Conclusions: The applications support authors to publish reproducible research predominantly with literate programming. Concerning readers, most applications provide user interfaces to inspect and manipulate the computational analysis. The next step is to investigate the gaps identified in this review, such as the costs publishers have to expect when hosting an application, the consideration of sensitive data, and impacts on the review process.

Principles for data analysis workflows

“…But judging a researcher's productivity by the metric of paper authorship can alter how and even whether research is performed [36]. Increasingly, academic journals are revisiting their publishing requirements [37] and raising their standards of reproducibility. This shift is bringing the data and programming methodologies that Research products can build off of content generated in either the Explore or the Refine Phase.…”

“…Peer-reviewed papers are of primary importance to the career and reputation of academic researchers [39], but the traditional format for such publications often does not take into account essential aspects of data-intensive analysis such as computational reproducibility [40]. Where strict requirements for reproducibility are not enforced by a given journal, researchers should nonetheless compile the supporting products that made our submitted manuscript possible-including relevant code and data, as well as the documentation of our computational tools and methodologies as described in the earlier sections of this paperinto a research compendium [37,[41][42][43]. The objective is to provide transparency to those who read or wish to replicate our academic publication and reproduce the workflow that led to our results.…”

A Badging System for Reproducibility and Replicability in Remote Sensing Research

“…Konkol et al [8] discuss ten infrastructures for reproducible practices. The authors perform analyses from the viewpoints of publishers, editors, authors, readers and reviewers, and librarians.…”

“…Notable FLOSS platforms for Remote Sensing research are R 7 (for statistical computing and production of highquality graphics), Python 8 (an interpreted, object-oriented, high-level programming language with dynamic semantics), GNU Octave 9 (an array-programming oriented language). The European Space Agency -ESA freely distributes SNAP -SeNtinel's Application Platform 10 .…”

Reproducibilidad en AGILE: Experiencias, logros y recomendaciones

“…El autor sugiere distintas aproximaciones para la mejora de la reproducción de flujos de trabajo geo-computacionales. Relacionado con la compartición de análisis computacionales, Konkol et al (2020) comparan exhaustivamente una serie de herramientas y aplicaciones para la publicación y ejecución de flujos de trabajos computacionales de forma reproducible.…”