Galaxy-Kubernetes integration: scaling bioinformatics workflows in the cloud

Moreno, Pablo; Pireddu, Luca; Roger, Pierrick; Goonasekera, Nuwan; Afgan, Enis; Beek, Marius van den; He, Shan; Larsson, Anders; Schober, Daniel; Ruttkies, Christoph; Johnson, David; Rocca-Serra, Philippe; Weber, R.; Gruening, Bjoern; Salek, Reza M.; Kale, Namrata; Perez‐Riverol, Yasset; Papatheodorou, Irene; Spjuth, Ola; Neumann, Steffen

doi:10.1101/488643

Cited by 19 publications

(15 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, through efforts within the Phe-noMeNal H2020 Project [26], Galaxy gained the ability to deploy inside Kubernetes in an automated fashion through the use of Helm Charts [27]. The Galaxy-stable Helm chart makes use of community maintained Galaxy container images (Docker-Galaxy-stable), which are used on other container orchestrators, and include support for SSH file transfer protocol (SFTP) access, database backends and even deployment/compatibility of HPC scheduling systems like Condor or SLURM for hybrid cloud setups.…”

Section: Galaxymentioning

confidence: 99%

Approaches for Containerized Scientific Workflows in Cloud Environments with Applications in Life Science

Spjuth¹,

Capuccini²,

Carone³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Containers are gaining popularity in life science research as they provide a solution for encompassing dependencies of provisioned tools, simplify software installations for end users and offer a form of isolation between processes. Scientific workflows are ideal for chaining containers into data analysis pipelines to aid in creating reproducible analyses. In this manuscript we review a number of approaches to using containers as implemented in the workflow tools Nextflow, Galaxy, Pachyderm, Argo, Kubeflow, Luigi and SciPipe, when deployed in cloud environments. A particular focus is placed on the workflow tool’s interaction with the Kubernetes container orchestration framework.

show abstract

Section: Galaxymentioning

confidence: 99%

Approaches for Containerized Scientific Workflows in Cloud Environments with Applications in Life Science

Spjuth¹,

Capuccini²,

Carone³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Galaxy (https://galaxyproject.org/) is a web application workflow environment written in Python, capable of distributing jobs among a plethora of batch schedulers (PBS, LSF, GoDocker, DRMAA based schedulers, etc. ), local machine (through containers or conda), and cloud providers (through Kubernetes and others). In the HPC case, Galaxy provides the flexibility to use either containers (Docker or Singularity) or directly Conda packages.…”

Section: Moving From Desktop Applications To Distributed Hpc and Cloumentioning

confidence: 99%

Scalable Data Analysis in Proteomics and Metabolomics Using BioContainers and Workflows Engines

Perez‐Riverol

Moreno

2019

Proteomics

Self Cite

View full text Add to dashboard Cite

The recent improvements in mass spectrometry instruments and new analytical methods are increasing the intersection between proteomics and big data science. In addition, bioinformatics analysis is becoming increasingly complex and convoluted, involving multiple algorithms and tools. A wide variety of methods and software tools have been developed for computational proteomics and metabolomics during recent years, and this trend is likely to continue. However, most of the computational proteomics and metabolomics tools are designed as single-tiered software application where the analytics tasks cannot be distributed, limiting the scalability and reproducibility of the data analysis. In this paper the key steps of metabolomics and proteomics data processing, including the main tools and software used to perform the data analysis, are summarized. The combination of software containers with workflows environments for large-scale metabolomics and proteomics analysis is discussed. Finally, a new approach for reproducible and large-scale data analysis based on BioContainers and two of the most popular workflow environments, Galaxy and Nextflow, is introduced to the proteomics and metabolomics communities.

show abstract

“…Kubernetes works with many container engines and enables automated deployment on cloud or on-premise clusters. One of the successful adoptions of Kubernetes in genomics is the Galaxy project [53]. However, Kubernetes does not reduce the deployment complexity for applications with complex workflows.…”

Section: Container Scalabilitymentioning

confidence: 99%

Computational Strategies for Scalable Genomics Analysis

Shi

Zhong

2019

Genes

View full text Add to dashboard Cite

The revolution in next-generation DNA sequencing technologies is leading to explosive data growth in genomics, posing a significant challenge to the computing infrastructure and software algorithms for genomics analysis. Various big data technologies have been explored to scale up/out current bioinformatics solutions to mine the big genomics data. In this review, we survey some of these exciting developments in the applications of parallel distributed computing and special hardware to genomics. We comment on the pros and cons of each strategy in the context of ease of development, robustness, scalability, and efficiency. Although this review is written for an audience from the genomics and bioinformatics fields, it may also be informative for the audience of computer science with interests in genomics applications.

show abstract

Galaxy-Kubernetes integration: scaling bioinformatics workflows in the cloud

Cited by 19 publications

References 10 publications

Approaches for Containerized Scientific Workflows in Cloud Environments with Applications in Life Science

Approaches for Containerized Scientific Workflows in Cloud Environments with Applications in Life Science

Scalable Data Analysis in Proteomics and Metabolomics Using BioContainers and Workflows Engines

Computational Strategies for Scalable Genomics Analysis

Contact Info

Product

Resources

About