The impact of Docker containers on the performance of genomic pipelines

Tommaso, Paolo Di; Palumbo, Emilio; Chatzou, Maria; Prieto, Pablo; Heuer, Michael; Notredame, Cédric

doi:10.7287/peerj.preprints.1171v1

Cited by 28 publications

(31 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It allows the seamless parallelization and deployment of any existing application with minimal development and maintenance overhead, irrespective of the original programming language. The built-in support for container technologies such as Docker 5 and Shifter 6 , along with the native integration with the Git tool and popular code-sharing platforms like GitHub, make it possible to precisely prototype self-contained computational workflows, maintain all variations over time and rapidly reproduce any former configuration one may need to re-use. These capabilities guarantee consistent results over time and across different computing platforms.…”

Section: Methodsmentioning

confidence: 99%

Using the Nextflow framework for reproducible in-silico omics analyses across clusters and clouds

Tommaso

Floden

Chatzou

et al. 2017

Preprint

Self Cite

View full text Add to dashboard Cite

Reproducibility has become one of biology’s most pressing issues. This impasse has been fuelled by the combined reliance on increasingly complex data analysis methods and the exponential growth of biological datasets. Nextflow is a pipeline orchestration tool that has been designed to ease deployment and guarantee reproducibility across platforms. It allows the seamless parallelization and deployment of any existing application with minimal development and maintenance overhead, irrespective of the original programming language. These capabilities guarantee consistent results over time and across different computing platforms.

show abstract

Section: Methodsmentioning

confidence: 99%

Using the Nextflow framework for reproducible in-silico omics analyses across clusters and clouds

Tommaso

Floden

Chatzou

et al. 2017

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Docker containers are shown to have better than, or equal performance as VMs. [19] Both forms of virtualization techniques introduce overhead in I/O-intensive workloads, especially in VMs, but introduce negligible CPU and memory overhead. For precision medicine pipelines the overhead of Docker containers will be negligible since these tend to be compute intensive and they typically run for several hours.…”

Section: Related Workmentioning

confidence: 99%

“…For precision medicine pipelines the overhead of Docker containers will be negligible since these tend to be compute intensive and they typically run for several hours. [19] Containers have also been proposed as a solution to improve experiment reproducibility, by ensuring that the data analysis tools are installed with the same responsibilities. [20]…”

Section: Related Workmentioning

confidence: 99%

Reproducible Data Analysis Pipelines for Precision Medicine

Fjukstad

Dumeaux

Hallett

et al. 2018

Preprint

View full text Add to dashboard Cite

Precision medicine brings the promise of more precise diagnosis and individualized therapeutic strategies from analyzing a cancer's genomic signature. Technologies such as high-throughput sequencing enable cheaper data collection at higher speed, but rely on modern data analysis platforms to extract knowledge from these high dimensional datasets. Since this is a rapidly advancing field, new diagnoses and therapies often require tailoring of the analysis. These pipelines are therefore developed iteratively, continuously modifying analysis parameters before arriving at the final results. To enable reproducible results it is important to record all these modifications and decisions made during the analysis process.We built a system, walrus, to support reproducible analyses for iteratively developed analysis pipelines. The approach is based on our experiences developing and using deep analysis pipelines to provide insights and recommendations for treatment in an actual breast cancer case. We designed walrus for the single servers or small compute clusters typically available for novel treatments in the clinical setting. walrus leverages software containers to provide reproducible execution environments, and integrates with modern version control systems to capture provenance of data and pipeline parameters.We have used walrus to analyze a patient's primary tumor and adjacent normal tissue, including subsequent metastatic lesions. Although we have used walrus for specialized analyses of whole-exome sequencing datasets, it is a general data analysis tool that can be applied in a variety of scientific disciplines.

show abstract

“…The container virtualization technology, represented by Docker, enables users to create a software runtime environment isolated from the host machine [3]. This technology that is getting popular also in the biomedical research domain is a promising method to solve the problem of installing software tools [4]. Along with the containers, using workflow description and execution frameworks such as those from the Galaxy project [5] or the Common Workflow Language (CWL) project [6] lowered the barrier to deploy the data analysis environment to a new computing environment.…”

mentioning

confidence: 99%

Accumulating computational resource usage of genomic data analysis workflow to optimize cloud computing instance selection

Ohta

Tanjo

Ogasawara

2018

Preprint

View full text Add to dashboard Cite

BackgroundContainer virtualization technologies such as Docker became popular in the bioinformatics domain as they improve portability and reproducibility of software deployment. Along with software packaged in containers, the workflow description standards Common Workflow Language also enabled to perform data analysis on multiple different computing environments with ease.These technologies accelerate the use of on-demand cloud computing platform which can scale out according to the amount of data. However, to optimize the time and the budget on a use of cloud, users need to select a suitable instance type corresponding to the resource requirements of their workflows. ResultsWe developed CWL-metrics, a system to collect runtime metrics of Docker containers and workflow metadata to analyze resource requirement of workflows. We demonstrated the analysis by using seven transcriptome quantification workflows on six instance types. The result showed instance type options of lower financial cost and faster execution time with required amount of computational resources. ConclusionsThe summary of resource requirements of workflow executions provided by CWL-metrics can help users to optimize the selection of cloud computing instance. The runtime metrics data also accelerate to share workflows among different workflow management frameworks. KeywordsHigh-throughput nucleotide sequencing, Cloud computing, Common Workflow Language BackgroundAccording to the improvement of DNA sequencing technology in accuracy and quantity, various sequencing methods are now available to measure different genomic features. Each method produces a massive amount of nucleotide sequence data that requires a different data processing approach [1].Bioinformatics researchers develop data analysis tools for each sequencing technique, and they publish implementations as open source software [2]. To

show abstract

The impact of Docker containers on the performance of genomic pipelines

Cited by 28 publications

References 0 publications

Using the Nextflow framework for reproducible in-silico omics analyses across clusters and clouds

Using the Nextflow framework for reproducible in-silico omics analyses across clusters and clouds

Reproducible Data Analysis Pipelines for Precision Medicine

Accumulating computational resource usage of genomic data analysis workflow to optimize cloud computing instance selection

Contact Info

Product

Resources

About