Capturing Data Provenance from Statistical Software

Abstract

We have created tools that automate one of the most burdensome aspects of documenting the provenance of research data: describing data transformations performed by statistical software. Researchers in many fields use statistical software (SPSS, Stata, SAS, R, Python) for data transformation and data management as well as analysis. The C2Metadata ("Continuous Capture of Metadata for Statistical Data") Project creates a metadata workflow paralleling the data management process by deriving provenance information from scripts used to manage and transform data. C2Metadata differs from most previous data provenance initiatives by documenting transformations at the variable level rather than describing a sequence of opaque programs. Command scripts for statistical software are translated into an independent Structured Data Transformation Language (SDTL), which serves as an intermediate language for describing data transformations. SDTL can be used to add variable-level provenance to data catalogues and codebooks and to create "variable lineages" for auditing software operations. Better data documentation makes research more transparent and expands the discovery and re-use of research data.

Towards Specificationless Monitoring of Provenance-Emitting Systems

“…Provenance data is either extracted from software systems after [43] or during their operation [24,27,47]. There is active work towards recording provenance information without instrumenting the system or process [3,5,18,39].…”

dtrackr: An R package for tracking the provenance of data

“…The general capability however fits into a broader context of other provenance or data pipeline research. This includes initiatives such as C2Metadata (Alter et al, 2021), which focus on a language independent representation of a data pipeline, and R packages such as targets (Landau, 2021) which focus on documenting pipeline code, and managing the execution of a pipeline, or RDataTracker which focusses on tracking the execution of a arbitrary R script (Lerner et al, 2018). dtrackr takes a more data oriented approach, which could be complementary, in which we remain agnostic to the detail of a data pipeline script or nature of its execution, but capture a subset of the transformations applied to data alongside the data itself, thereby documenting the data state as it is being manipulated.…”