Building Containerized Workflows Using the BioDepot-Workflow-Builder

Hung, Ling‐Hong; Hu, Jiaming; Meiss, Trevor; Ingersoll, Alyssa; Lloyd, Wes; Kristiyanto, Daniel; Xiong, Yuguang; Sobie, Eric A.; Yeung, Ka Yee

doi:10.1016/j.cels.2019.08.007

Cited by 18 publications

(19 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We previously reported optimizations of the implementation that reduce the total execution time to 3.5 hours when using 16 threads on the same type of cloud server 5 . As expected, the longest step is the CPU-intensive alignment which takes 19.5 hours in the original 14 and 2.5 hours in the optimized implementation 5 . We benchmarked the serverless version of the workflow and observed that transcript counts can be obtained in 6 minutes at a cost of $3.85 (see Methods for implementation details, benchmarking code and cost calculations ).…”

supporting

confidence: 75%

“…As a case study to demonstrate this novel use of serverless computing, we present an RNA-seq workflow using unique molecular identifiers (UMI) to obtain de-duped transcript counts 12 which are then processed by edgeR 13 to obtain a list of genes that were differentially expressed upon treatment with different combinations of drugs 14 . For the analysis of this dataset, we invoked 1752 function instances to reduce the alignment time from 19.5 hours to 1 minute when using the most performant configuration on AWS (Table S2).…”

mentioning

confidence: 99%

“…The resulting alignments are merged and de-duped using Unique Molecular Identifiers (UMIs) to compute the observed counts for each transcript. The original workflow 14 took 29.5 hours to obtain transcript counts using a virtual cloud server (AWS EC2 instance m4.4xlarge). We previously reported optimizations of the implementation that reduce the total execution time to 3.5 hours when using 16 threads on the same type of cloud server 5 .…”

mentioning

confidence: 99%

“…We developed a containerized graphical front-end for the serverless version of the UMI RNA-seq workflow by extending our Biodepot-workflow-builder framework 14 (see Figure 1 ), that can run from a laptop, desktop, or cloud server. The entire setup is automated with the user needing only to enter the credentials for the cloud account being used for the serverless instances, and the location of the files that are to be processed.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Accessible and interactive RNA sequencing analysis using serverless computing

Hung

Niu

Lloyd

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

We have used serverless AWS Lambda functions to align 640 million reads in less than 3 minutes, a speed-up of 500x over the single-threaded implementation. Using a hybrid cloud architecture and software modified to optimize disk transfers, an entire RNA sequencing workflow transforming multiplexed reads to transcript counts that originally took 29 hours can be completed in 18 minutes. This is a 100x improvement over the original single threaded implementation and 12x faster than an optimized cloud server-based implementation using 16 threads. The total cost of the analyses is $2.82 for 96 wells or 3 cents per multiplexed sample.This approach can be used for human datasets that are generated for single experiments and does not rely on processing large numbers of samples to achieve the performance gains. The workflow is publicly available under a M.I.T. license (https://github.com/BioDepot/RNA-seqlambda).

show abstract

supporting

confidence: 75%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Accessible and interactive RNA sequencing analysis using serverless computing

Hung

Niu

Lloyd

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Containerized workflows are used to decouple complicated processes into manageable software programs. Cloud-based data pipelines and containerized applications simplify workflow management and execution (Hasham et al, 2018;Marcus et al, 2007;Hung et al, 2019;Abe et al;Jansen et al, 2020). Cloud-controlled IoT systems enable researchers to take advantage of machine learning to optimize scientific studies.…”

Section: Introductionmentioning

confidence: 99%

Internet of Things Architecture for Cellular Biology

Parks

Voitiuk

Geng

et al. 2021

Preprint

View full text Add to dashboard Cite

New cell culture techniques have led to complex tissue models in biological experiments. For example, 3-D cerebral organoids provide a more realistic model of the human cortical tissue. However, these cell culture experiments are restricted by high costs and limited labor. A massively scalable and cost efficient platform for tissue experiments would benefit genomics, neuroscience, and translational medicine by enabling advanced high throughput tissue screens. Cloud computing and the Internet of Things (IoT) provide new tools for managing multiple experiments in parallel that are remotely controlled through automation. We introduce a cloud-based IoT architecture that takes advantage of these tools to offer an environment where researchers can run thousands of cell culture experiments at once. This technology allows studies with cell cultures to be performed at scales far beyond a single lab setting, democratizing access to advanced tissue models and enabling new avenues of research.

show abstract