The Dockstore: enabling modular, community-focused sharing of Docker-based genomics tools and workflows

Abstract

As genomic datasets continue to grow, the feasibility of downloading data to a local organization and running analysis on a traditional compute environment is becoming increasingly problematic. Current large-scale projects, such as the ICGC PanCancer Analysis of Whole Genomes (PCAWG), the Data Platform for the U.S. Precision Medicine Initiative, and the NIH Big Data to Knowledge Center for Translational Genomics, are using cloud-based infrastructure to both host and perform analysis across large data sets. In PCAWG, over 5,800 whole human genomes were aligned and variant called across 14 cloud and HPC environments; the processed data was then made available on the cloud for further analysis and sharing. If run locally, an operation at this scale would have monopolized a typical academic data centre for many months, and would have presented major challenges for data storage and distribution. However, this scale is increasingly typical for genomics projects and necessitates a rethink of how analytical tools are packaged and moved to the data. For PCAWG, we embraced the use of highly portable Docker images for encapsulating and sharing complex alignment and variant calling workflows across highly variable environments. While successful, this endeavor revealed a limitation in Docker containers, namely the lack of a standardized way to describe and execute the tools encapsulated inside the container. As a result, we created the Dockstore ( https://dockstore.org), a project that brings together Docker images with standardized, machine-readable ways of describing and running the tools contained within. This service greatly improves the sharing and reuse of genomics tools and promotes interoperability with similar projects through emerging web service standards developed by the Global Alliance for Genomics and Health (GA4GH).

Robust Cross-Platform Workflows: How Technical and Scientific Communities Collaborate to Develop, Test and Share Best Practices for Data Analysis

“…Incompatible versions of interacting tools would disrupt the workflow as a whole. This motivated Dockstore to shift from dynamic image creation with Ansible to ready-prepared Docker images [30]. The exact specification of versions from snapshot.debian.org, Bioconda, or unspecified versions of two tools together in the same release of a distribution is expected to overcome this difficulty.…”

“…To the rescue come registries of software like Bio.Tools [17], and the resource identification initiative (https://scicrunch.org/resources) expands the same concept well beyond software. For sharing whole workflows, myExperiment (http://myexperiment.org/) is a well-established repository used by multiple workflow systems and research communities [10], complementing more specific workflow repositories like CWL Viewer [33] (https://view.commonwl.org/), Galaxy's ToolShed (https:// galaxyproject.org/toolshed/workflow-sharing/), and Dockstore [30].…”

nf-core: Community curated bioinformatics pipelines

“…As the usage of workflow management tools spreads, an increasing number of tertiary tools are tying into the ecosystem. The nf-core analysis pipelines are at the forefront of this, collaborating with initiatives such as bio.tools 28 , the GA4GH-compliant Dockstore 29 , and with plans to work together with the Biocontainers 18 project in the future to further simplify software packaging.…”

Practical Computational Reproducibility in the Life Sciences

“…Container registries provide an authoritative mapping from a container name (e.g., https://quay.io/repository/biocontainers/hisat2?tag=2.1.0-py27pl5.22.0_0&tab=tags) to a specific container image. We register containers with Quay (https://quay.io), and since we build standard open container initiative (OCI) containers (https://www.opencontainers.org), other registries (e.g., DockerHub, Cook, 2017; BioContainers, da Veiga Leprevost et al, 2017; or Dockstore, O’Connor et al, 2017) can also be used. Finally, in addition to creating containers for single Bioconda packages, it is possible to automatically create containers for combinations of packages.…”