Scalable Workflows and Reproducible Data Analysis for Genomics

Strozzi, Francesco; Janssen, Roel; Wurmus, Ricardo; Crusoe, Michael R.; Githinji, George; Tommaso, Paolo Di; Belhachemi, Dominique; Möller, Steffen; Smant, G.; Ligt, Joep de; Prins, Pjotr

doi:10.1007/978-1-4939-9074-0_24

Cited by 32 publications

(32 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…ics data analysis workflow with flexibility, extensibility and reproducibility [39,40,41,42,43,44]. Herein, we used Nextflow to encapsulate the seven data processing pipelines so that users can directly call different pipelines through a uniform set of APIs.…”

Section: Nextflow Pipeline Design and Featuresmentioning

confidence: 99%

Comparison of High-Throughput Single-Cell RNA Sequencing Data Processing Pipelines

Gao

Ling

Tang

et al. 2020

Preprint

View full text Add to dashboard Cite

With the development of single-cell RNA sequencing (scRNA-seq) technology, it has become possible to perform large-scale transcript profiling for tens of thousands of cells in a single experiment. Many analysis pipelines have been developed for data generated from different high-throughput scRNA-seq platforms, bringing a new challenge to users to choose a proper workflow that is efficient, robust and reliable for a specific sequencing platform. Moreover, as the amount of public scRNA-seq data has increased rapidly, integrated analysis of scRNAseq data from different sources has become increasingly popular. However, it remains unclear whether such integrated analysis would be biased if the data were processed by different upstream pipelines. In this study, we encapsulated seven existing high-throughput scRNA-seq data processing pipelines with Nextflow, a general integrative workflow management framework, and evaluated their performances in terms of running time, computational resource consumption, and data processing consistency using nine public datasets generated from five different high-throughput scRNA-seq platforms. Our work provides a useful guideline for the selection of scRNA-seq data processing pipelines based on their performances on different real datasets. In addition, these guidelines can serve as a performance evaluation framework for future developments in high-throughput scRNA-seq data processing.Since the emergence of the first single-cell RNA sequencing (scRNA-seq) platform [1], many research achievements have been made at the cellular and subcellular levels with unprecedented resolutions with the aid of this technology. Recent advances in microfluidics and next generation sequencing (NGS) have further increased the efficiency and throughput of scRNA-seq, enabling more cells to be identified and the expression information of more genes for each cell to be quantified simultaneously [2,3,4]. From microcapillary pipettes in 2012 [5,6] and nanoliter droplet-based microfluidic chips in 2015 [7,8] to the latest liquid barcoding combined with split-pool methods [9], the number of cells analysed in parallel has increased from several to tens of thousands [10], bringing new challenges for the processing of a large amount of barcoded NGS data for efficient and accurate quantification of the transcript information of single cells.To meet the needs of high-throughput scRNA-seq data processing, several pipelines that integrate multiple functions have been developed. As one of the first high-throughput scRNA-seq platforms, Drop-seq [7] was introduced in 2015. Moreover, Drop-seq-tools in combination with Picard tools were introduced and provided a user-friendly Application Programming Interface (API) to process the data from Drop-seq. Later in 2017, three additional pipelines were published including Cell Ranger [11], umis [12] and UMItools [13]. Cell Ranger [11] was developed along with the widely adopted 10X Genomics platform which can process multiple biological samples separated by sample barcodes in paral...

show abstract

Section: Nextflow Pipeline Design and Featuresmentioning

confidence: 99%

Comparison of High-Throughput Single-Cell RNA Sequencing Data Processing Pipelines

Gao

Ling

Tang

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Reproducibility in biomedical analyses has become important recently (Grüning et al, 2018; Strozzi et al, 2019). For workflows, this means that anyone should be able to reconstruct the exact workflow run, including the correct sequence of steps, the actual commands, the runtime parameters and options, and the handling of data, e.g., chunking for parallelization, to reach the exact same conclusions despite differences in hardware, operating systems, and software dependencies.…”

Section: Resultsmentioning

confidence: 99%

Design considerations for workflow management systems use in production genomics research and the clinic

Ahmed

Allen

Bhat

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: The changing landscape of genomics research and clinical practice has created a need for computational pipelines capable of efficiently orchestrating complex analysis stages while handling large volumes of data across heterogeneous computational environments. Workflow Management Systems (WfMSs) are the software components employed to fill this gap. Results: This work provides an approach and systematic evaluation of key features of popular bioinformatics WfMSs in use today: Nextflow, CWL, and WDL and some of their executors, along with Swift/T, a workflow manager commonly used in high-scale physics applications. We employed two use cases: a variant-calling genomic pipeline and a scalability-testing framework, where both were run locally, on an HPC cluster, and in the cloud. This allowed for evaluation of those four WfMSs in terms of language expressiveness, modularity, scalability, robustness, reproducibility, interoperability, ease of development, along with adoption and usage in research labs and healthcare settings. This article is trying to answer, "which WfMS should be chosen for a given bioinformatics application regardless of analysis type?". Conclusions: The choice of a given WfMS is a function of both its intrinsic language and engine features. Within bioinformatics, where analysts are a mix of dry and wet lab scientists, the choice is also governed by collaborations and adoption within large consortia and technical support provided by the WfMS team/community. As the community and its needs continue to evolve along with computational infrastructure, WfMSs will also evolve, especially those with permissive licenses that allow commercial use. In much the same way as the dataflow paradigm and containerization are now well understood to be very useful in bioinformatics applications, we will continue to see innovations of tools and utilities for other purposes, like big data technologies, interoperability, and provenance.

show abstract

“…Workflow management systems are developed for robust and easy implementation of computational pipelines; nevertheless, they differ significantly in terms of workflows, definitions, job scheduling, and features (5,6,7,8). For example, Snakemake uses a "pullbased" strategy to check for specific output files and schedule jobs accordingly (5,6), whereas Nextflow uses a "push-based" scheme in which a "process" defined in the workflow pushes its outputs to downstream "processes" (7,24). The SciPipe (5) workflow library is written in the GO language; similar to Nextflow it implements dataflow based task scheduling.…”

Section: Discussionmentioning

confidence: 99%

pyrpipe: a python package for RNA-Seq workflows

Singh

Seetharam

et al. 2020

Preprint

View full text Add to dashboard Cite

Implementing RNA-Seq analysis pipelines is challenging as data gets bigger and more complex. With the availability of terabytes of RNA-Seq data and continuous development of analysis tools, there is a pressing requirement for frameworks that allow for fast and efficient development, modification, sharing and reuse of workflows. Scripting is often used, but it has many challenges and drawbacks. We have developed a python package, python RNA-Seq Pipeliner (pyrpipe) that enables straightforward development of flexible, reproducible and easy-to-debug computational pipelines purely in python, in an object-oriented manner. pyrpipe provides high level APIs to popular RNA-Seq tools. Pipelines can be customized by integrating new python code, third-party programs, or python libraries. Researchers can create checkpoints in the pipeline or integrate pyrpipe into a workflow management system, thus allowing execution on multiple computing environments. pyrpipe produces detailed analysis, and benchmark reports which can be shared or included in publications. pyrpipe is implemented in python and is compatible with python versions 3.6 and higher. All source code is available at https://github.com/urmi-21/pyrpipe; the package can be installed from the source or from PyPi (https://pypi.org/project/ pyrpipe).

show abstract

Scalable Workflows and Reproducible Data Analysis for Genomics

Cited by 32 publications

References 33 publications

Comparison of High-Throughput Single-Cell RNA Sequencing Data Processing Pipelines

Comparison of High-Throughput Single-Cell RNA Sequencing Data Processing Pipelines

Design considerations for workflow management systems use in production genomics research and the clinic

pyrpipe: a python package for RNA-Seq workflows

Contact Info

Product

Resources

About