Versioning Data Is About More than Revisions: A Conceptual Framework and Proposed Principles

Abstract

A dataset, small or big, is often changed to correct errors, apply new algorithms, or add new data (e.g., as part of a time series), etc. In addition, datasets might be bundled into collections, distributed in different encodings or mirrored onto different platforms. All these differences between versions of datasets need to be understood by researchers who want to cite the exact version of the dataset that was used to underpin their research. Failing to do so reduces the reproducibility of research results. Ambiguous identification of datasets also impacts researchers and data centres who are unable to gain recognition and credit for their contributions to the collection, creation, curation and publication of individual datasets. Although the means to identify datasets using persistent identifiers have been in place for more than a decade, systematic data versioning practices are currently not available. In this work, we analysed 39 use cases and current practices of data versioning across 33 organisations. We noticed that the term 'version' was used in a very general sense, extending beyond the more common understanding of 'version' to refer primarily to revisions and replacements. Using concepts developed in software versioning and the Functional Requirements for Bibliographic Records (FRBR) as a conceptual framework, we developed six foundational principles for versioning of datasets: Revision, Release, Granularity, Manifestation, Provenance and Citation. These six principles provide a high-level framework for guiding the consistent practice of data versioning and can also serve as guidance for data centres or data providers when setting up their own data revision and version protocols and procedures.

The Oceans 2.0/3.0 Data Management and Archival System

“…This versioning approach is consistent with recommendations from the Research Data Alliance (RDA) Data Citation Working Group (Rauber et al, 2015) and the RDA Data Versioning Working Group (Principles 1, 5, and 6 of Klump et al, 2021). As new standards and best practices emerge from the research data community, ONC will continue to improve these frameworks.…”

Data-centric AI: Perspectives and Challenges

“…It is a communication protocol with many benefits, including software reuse, information extraction, data summarization, etc. Another noteworthy task is data version control [17], which records the changes to a dataset during its lifecycle to achieve traceable error reporting and enable reproducible results.…”

Challenges to sharing sample metadata in computational genomics

“…Despite clear mutability, metadata tables are often treated as static. Version control is well established for code and has a diverse and multifaceted history for data as well ( Klump et al, 2021 ), but the question of versioning metadata specifically is distinct. A common strategy for versioning sample metadata is to use tools built for code versioning, such as git.…”