Approaches for containerized scientific workflows in cloud environments with applications in life science

Abstract

Containers are gaining popularity in life science research as they encompass all dependencies of provisioned tools and simplifies software installations for end users, as well as offering a form of isolation between processes. Scientific workflows are ideal to chain containers into data analysis pipelines to sustain reproducible science. In this manuscript we review the different approaches to use containers inside the workflow tools Nextflow, Galaxy, Pachyderm, Luigi, and SciPipe when deployed in cloud environments. A particular focus is placed on the workflow tool's interaction with the Kubernetes container orchestration framework.

Compi: a framework for portable and reproducible pipelines

“…Such a Docker runner would allow to follow an image-per-task execution pattern, where each task is executed using a different container image ( Spjuth et al, 2018 ). An example of this execution pattern can be found in the GenomeFastScreen pipeline ( https://sing-group.org/compihub/explore/5e2eaacce1138700316488c1 ), although in this case “docker run” commands are included in each task rather than provided in a runners file for the sake of simplicity.…”

“…As noted in the previous section, this is another way to deal with dependency management, as such Docker images contain all the dependencies required by the pipeline. Pipelines distributed in this way follow an image-per-pipeline execution pattern in which all tasks are executed using the same image container ( Spjuth et al, 2018 ) ( Fig. 5C ), and can even be run using Docker-compatible container technologies such as Singularity.…”

“…Despite the existence of such remarkable workflow management systems, scientists with basic scripting skills (e.g., able to create shell scripts invoking command-line tools) but lacking advanced programming skills (e.g., knowledge of programming languages such as Python or Go), are usually overwhelmed due to the high complexity of these systems, and could be hampered to use or create their own workflows. In this sense, tools such as Bpipe (Sadedin, Pope & Oshlack, 2012) help to assemble shell scripts into workflows to aid in job automation, logging and reproducibility.…”

polishCLR: A Nextflow Workflow for Polishing PacBio CLR Genome Assemblies

“…Increasing interest in workflow development systems that track data and software provenance, enable scalability and reproducibility, and re-entrant code ( Wratten et al 2021 ) have led to the development of several workflow languages, largely inspired by GNU Make ( Stallman and McGrath 1991 ; Köster and Rahmann 2012 ; Amstutz et al 2016 ). Nextflow is a Domain Specific Language ( Di Tommaso et al 2017 ) that currently leads workflow systems in terms of ease of scripting and submitting to cloud computing resources ( Fjukstad and Bongo 2017 ; Leipzig 2017 ; Spjuth et al 2020 ; Jackson et al 2021 ). A key benefit of Nextflow compared with earlier workflow languages is being able to submit jobs to a local machine, an HPC, or cloud-based compute environments.…”