SciPipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines

Abstract

Background The complex nature of biological data has driven the development of specialized software tools. Scientific workflow management systems simplify the assembly of such tools into pipelines, assist with job automation, and aid reproducibility of analyses. Many contemporary workflow tools are specialized or not designed for highly complex workflows, such as with nested loops, dynamic scheduling, and parametrization, which is common in, e.g., machine learning. Findings SciPipe is a workflow programming library implemented in the programming language Go, for managing complex and dynamic pipelines in bioinformatics, cheminformatics, and other fields. SciPipe helps in particular with workflow constructs common in machine learning, such as extensive branching, parameter sweeps, and dynamic scheduling and parametrization of downstream tasks. SciPipe builds on flow-based programming principles to support agile development of workflows based on a library of self-contained, reusable components. It supports running subsets of workflows for improved iterative development and provides a data-centric audit logging feature that saves a full audit trace for every output file of a workflow, which can be converted to other formats such as HTML, TeX, and PDF on demand. The utility of SciPipe is demonstrated with a machine learning pipeline, a genomics, and a transcriptomics pipeline. Conclusions SciPipe provides a solution for agile development of complex and dynamic pipelines, especially in machine learning, through a flexible application programming interface suitable for scientists used to programming or scripting.

pyrpipe: a python package for RNA-Seq workflows

“…Workflow management systems are developed for robust and easy implementation of computational pipelines; nevertheless, they differ significantly in terms of workflows, definitions, job scheduling, and features (5,6,7,8). For example, Snakemake uses a "pullbased" strategy to check for specific output files and schedule jobs accordingly (5,6), whereas Nextflow uses a "push-based" scheme in which a "process" defined in the workflow pushes its outputs to downstream "processes" (7,24). The SciPipe (5) workflow library is written in the GO language; similar to Nextflow it implements dataflow based task scheduling.…”

“…A major challenge in bioinformatics analysis of RNA-Seq datasets is implementing data processing pipelines in an efficient, modular, and reproducible manner (4,5,6,7). A majority of existing bioinformatics tools are standalone linux programs, executed via the shell; writing bioinformatic pipelines as shell, perl, or python scripts is a very common practice among bioinformaticians.…”

MasterOfPores: A Workflow for the Analysis of Oxford Nanopore Direct RNA Sequencing Datasets

“…Workflow management systems together with Linux containers offer a solution to efficiently analyze large scale datasets in a highly reproducible, scalable and parallelizable manner. During the last decade, an increasing interest in the field has led to the development of different programs such as Snakemake (Köster andRahmann, 2012), NextFlow (Di Tommaso et al, 2017), Galaxy (Afgan et al, 2018), SciPipe (Lampa et al, 2019) or GenPipes (Bourgey et al, 2019), among others. These tools enable the prototyping and deployment of pipelines by abstracting computational processes and representing pipelines as directed graphs, in which nodes represent tasks to be executed and edges represent either data flow or execution dependencies between different tasks.…”

A Serverless Tool for Platform Agnostic Computational Experiment Management

“…While Nipype is a widely used tool in neuroimaging and has many readily-defined interfaces available for researchers, the others require non-insignificant development to describe interfaces for common neuroimaging applications such as FSL ( Jenkinson et al, 2012 ) or MRtrix ( Tournier and Calamante, 2012 ). PSOM ( Bellec et al, 2012 ) and Scipipe ( Lampa et al, 2018 ) are functionally similar to Nipype but have been developed for GNU Octave/MATLAB and Golang, respectively. Several domains have more specialized tools which accomplish similar feats in their area of interest.…”