Container-based bioinformatics with Pachyderm

Novella, Jon Ander; Khoonsari, Payam Emami; Herman, Stephanie; Whitenack, Daniel; Capuccini, Marco; Burman, Joachim; Kultima, Kim; Spjuth, Ola

doi:10.1093/bioinformatics/bty699

Cited by 47 publications

(34 citation statements)

References 35 publications

(36 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Harnessing Docker To Create a Reproducible Runtimementioning

confidence: 99%

“…PharmacoGx integrates seamlessly with CWL, as we leverage CWL's Docker capabilities to containerize the package and run all of our pipelines in an isolated environment 20 . Docker is a tool that allows for PharmacoGx to be uniformly deployed with all software dependencies, in a containerized runtime environment where all of our computations are performed and PSets are produced 20 . The Docker container is invoked upon CWL workflow execution, where all the input files for a given pipeline become mounted into the container and all output files produced in the isolated environment are recovered into a local environment 22,23 .…”

Section: Harnessing Docker To Create a Reproducible Runtimementioning

confidence: 99%

See 1 more Smart Citation

Creating reproducible pharmacogenomic analysis pipelines

Mammoliti

Смирнов

Safikhani

et al. 2019

Preprint

View full text Add to dashboard Cite

The field of Pharmacogenomics presents great challenges for researchers that are willing to make their studies reproducible and shareable. This is attributed to the generation of large volumes of high-throughput multimodal data, and the lack of standardized workflows that are robust, scalable, and flexible to perform large-scale analyses. To address this issue, we developed pharmacogenomic workflows in the Common Workflow Language to process two breast cancer datasets in a reproducible and transparent manner. Our pipelines combine both pharmacological and molecular profiles into a portable data object that can be used for future analyses in cancer research. Our data objects and workflows are shared on Harvard Dataverse and Code Ocean where they have been assigned a unique Digital Object Identifier, providing a level of data provenance and a persistent location to access and share our data with the community.

show abstract

Section: Harnessing Docker To Create a Reproducible Runtimementioning

confidence: 99%

Section: Harnessing Docker To Create a Reproducible Runtimementioning

confidence: 99%

Creating reproducible pharmacogenomic analysis pipelines

Mammoliti

Смирнов

Safikhani

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The cloud infrastructure of PhenoMeNal is based upon containers that are deployed in a Kubernetes environment. Deployment is managed by KubeNow, which is developed by the PhenoMeNal team in order to simplify managing the deployment, including storage, network and other required services [20,35]. Orchestration is handled by using Helm charts.…”

Section: Cloud E-infrastructurementioning

confidence: 99%

“…Galaxy facilitates collaborative exchange, reproducibility and traceability of data analysis by enabling users to share entire workflows and analysis histories [19]. In addition to Galaxy, programmatic executable notebooks (Jupyter) and the programmatic interfaces Luigi and Pachyderm are also supported [20].…”

Section: Scientific Workflowsmentioning

confidence: 99%

PhenoMeNal: Processing and analysis of Metabolomics data in the Cloud

Peters

Bradbury

Bergmann

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Background: Metabolomics is the comprehensive study of a multitude of small molecules to gain insight into an organism's metabolism. The research field is dynamic and expanding with applications across biomedical, biotechnological and many other applied biological domains. Its computationally-intensive nature has driven requirements for open data formats, data repositories and data analysis tools. However, the rapid progress has resulted in a mosaic of independent -and sometimes incompatible -analysis methods that are difficult to connect into a useful and complete data analysis solution. Findings: The PhenoMeNal (Phenome and Metabolome aNalysis) e-infrastructure provides a complete, workflow-oriented, interoperable metabolomics data analysis solution for a modern infrastructure-as-a-service (IaaS) cloud platform. PhenoMeNal seamlessly integrates a wide array of existing open source tools which are tested and packaged as Docker containers through the project's continuous integration process and deployed based on a kubernetes orchestration framework. It also provides a number of standardized, automated and published analysis workflows in the user interfaces Galaxy, Jupyter, Luigi and Pachyderm. Conclusions: PhenoMeNal constitutes a keystone solution in cloud infrastructures available for metabolomics. It provides scientists with a ready-to-use, workflow-driven, reproducible and shareable data analysis platform harmonizing the software installation and configuration through user-friendly web interfaces. The deployed cloud environments can be dynamically scaled to enable large-scale analyses which are interfaced through standard data formats, versioned, and have been tested for reproducibility and interoperability. The flexible implementation of PhenoMeNal allows easy adaptation of the infrastructure to other application areas and 'omics research domains.

show abstract

“…The scheduling of pipeline workers is determined by the resource utilisation status of the workers nodes and the resource requests that are made for each pipeline, making it able to efficiently distribute workloads. Recently, work to integrate and demonstrate Pachyderm in bioinformatics was published [22] . Luigi Luigi ( https://github.com/spotify/luigi ) is an open-source Python module for defining and running batch-like workflows.…”

Section: Galaxymentioning

confidence: 99%

Approaches for containerized scientific workflows in cloud environments with applications in life science

Spjuth

Capuccini

Carone

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Containers are gaining popularity in life science research as they encompass all dependencies of provisioned tools and simplifies software installations for end users, as well as offering a form of isolation between processes. Scientific workflows are ideal to chain containers into data analysis pipelines to sustain reproducible science. In this manuscript we review the different approaches to use containers inside the workflow tools Nextflow, Galaxy, Pachyderm, Luigi, and SciPipe when deployed in cloud environments. A particular focus is placed on the workflow tool's interaction with the Kubernetes container orchestration framework.

show abstract

Container-based bioinformatics with Pachyderm

Abstract: Supplementary data is available at Bioinformatics online.

Cited by 47 publications

References 35 publications

Creating reproducible pharmacogenomic analysis pipelines

Creating reproducible pharmacogenomic analysis pipelines

PhenoMeNal: Processing and analysis of Metabolomics data in the Cloud

Approaches for containerized scientific workflows in cloud environments with applications in life science

Contact Info

Product

Resources

About