FAIR data pipeline: provenance-driven data management for traceable scientific workflows

Abstract

Modern epidemiological analyses to understand and combat the spread of disease depend critically on access to, and use of, data. Rapidly evolving data, such as data streams changing during a disease outbreak, are particularly challenging. Data management is further complicated by data being imprecisely identified when used. Public trust in policy decisions resulting from such analyses is easily damaged and is often low, with cynicism arising where claims of ‘following the science’ are made without accompanying evidence. Tracing the provenance of such decisions back through open software to primary data would clarify this evidence, enhancing the transparency of the decision-making process. Here, we demonstrate a Findable, Accessible, Interoperable and Reusable (FAIR) data pipeline. Although developed during the COVID-19 pandemic, it allows easy annotation of any data as they are consumed by analyses, or conversely traces the provenance of scientific outputs back through the analytical or modelling source code to primary data. Such a tool provides a mechanism for the public, and fellow scientists, to better assess scientific evidence by inspecting its provenance, while allowing scientists to support policymakers in openly justifying their decisions. We believe that such tools should be promoted for use across all areas of policy-facing research. This article is part of the theme issue ‘Technical challenges of modelling real-life epidemics and examples of overcoming these’.

Estimation of age-stratified contact rates during the COVID-19 pandemic using a novel inference algorithm

“…In addition to the results presented here, this algorithm provided parameter estimates for a non-age-structured COVID-19 model used by Mitchell et al . [ 42 ] to demonstrate the ability of the FAIR Data Pipeline to create traceable scientific workflows, transparently linking model outputs to inputs, that are critical for policy focused analysis. Application of ABC-MBP to the age-structured models, and publicly available COVID-19 data, studied here demonstrated a good model fit based on the inferred parameter values, and yielded several new insights.…”

Automation of Experimental Workflows for High Throughput Robotic Cultivations

“…To tackle this issue, robotic cultivation platforms optimally operated with computational tools can serve as generators of high-information content data. Such platforms rely on the integration of numerous interconnected devices, such as liquid handling stations, parallel cultivation systems and robots and their ability to generate the required data and provide provenance details regarding the rationale behind experimental design and the workflows used for conducting the experiments (Mitchell et al, 2022). A workflow management system (WMS) for automatically scheduling and executing the experimental and computational tasks not only increases the degree of automation but also contributes to implementing findable, accessible, interoperable and re-usable (FAIR) data principles (Wilkinson et al, 2016).…”

The Royal Society RAMP modelling initiative

“…The led to some significant codes developed from scratch: from Mathematics in Cambridge, a compartment model code/framework, PyRoss [3]; from Physics in Durham, an agent-based model for the UK at the scale of individual people, JUNE [4]; from Edinburgh, a datadriven R-calculation and medium-term prediction code [5]; and from Computer Science at Strathclyde, an application of process calculus to couple a within-host immune-response model to a population-level epidemic [6]. A pan-Scotland collaboration of animal disease modellers, the Scottish COVID-19 Response Consortium, produced an entire suite of codes and linking data pipelines [7].…”