The impact of Docker containers on the performance of genomic pipelines

Tommaso, Paolo Di; Palumbo, Emilio; Chatzou, Maria; Prieto, Pablo; Heuer, Michael; Notredame, Cédric

doi:10.7287/peerj.preprints.1171

Cited by 6 publications

(9 citation statements)

References 3 publications

(3 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Resultsmentioning

confidence: 99%

“…Furthermore, we are investigating the feasibility of containerization of the i2b2 platform, as Docker containers have been reported to be particularly useful to instantiate infrastructure for scientific computing. 23,24 We have limited the scope of the project to the PostgreSQL database, and several i2b2 sites are known to prefer other relational databases such as Oracle and Microsoft SQL. This is due to the proprietary nature of the other databases, which prohibits their distribution in open source.…”

Section: Limitations and Future Workmentioning

confidence: 99%

See 1 more Smart Citation

Automating Installation of the Integrating Biology and the Bedside (i2b2) Platform

Wagholikar

Mendis

Dessai

et al. 2018

Biomed Inform Insights

View full text Add to dashboard Cite

Informatics for Integrating Biology and the Bedside (i2b2) is an open source clinical data analytics platform used at more than 150 institutions for querying patient data. An i2b2 installation (called hive) comprises several i2b2 cells that provide different functionalities. Given the complex architecture of i2b2 installation, creating a working installation of the platform is challenging for new users. This is despite the availability of extensive documentation for i2b2 and access to a large and active mailing list community of i2b2 users. To address this problem, we have created an automated installation package, called i2b2-quickstart, which automatically downloads the latest i2b2 source code and dependencies, and compiles and configures the i2b2 cells to create a functional i2b2 hive installation. This package will serve as a convenient starting point and reference implementation that will facilitate researchers in the installation and exploration of the i2b2 platform.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Limitations and Future Workmentioning

confidence: 99%

Automating Installation of the Integrating Biology and the Bedside (i2b2) Platform

Wagholikar

Mendis

Dessai

et al. 2018

Biomed Inform Insights

View full text Add to dashboard Cite

show abstract

“…We describe the problems solved, analyze container execution times, discuss the limitations of the current solution and give an outlook to future developments. 21 http://mongodb.com/, as of 22.02.2018…”

Section: Discussionmentioning

confidence: 99%

“…Those in turn will be written into a JavaScript Object Notation (JSON) file inside a host mapped directory linked to a NAS drive. Going from there, the system transfers the file contents to a document-oriented database (such as MongoDB 21 ) and finally incorporates the evaluation scores into the user's instance-detail view.…”

Section: ) Node Execution Typementioning

confidence: 99%

“…This greatly reduces overhead, such as system setup, disk space and boot up time, which makes them highly portable [18], [19]. In terms of performance, a container only adds negligible overhead compared to native execution [20], [21], [22], [23]. Originally, containers were mainly used for software development and deployment, however there is also a trend that tends to shift thinking away from traditional patterns towards more creative ways of how to further exploit the container technology in various domains, such as facilitating reproducibility in research [24], [25].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Distributed Container-Based Evaluation Platform for Private/Large Datasets

Eggel

Schaer

Müller

2018

2018 17th International Symposium on Parallel and Distributed Computing (ISPDC)

View full text Add to dashboard Cite

The rise of big data and artificial intelligence techniques such as deep learning has lead to an exponential increase in stored data in various fields, including medical imaging, genetics and financial trading. Sharing these increasing amounts of data for research is challenging, as privacy risks increase with the increased size of data. Physically moving very large datasets to researchers is inconvenient, as download or sending physical hard disks are not optimal. Research on sensitive data is often not possible, as sharing is not legal. The popularity of containerbased technologies such as Docker has revolutionized the way applications are deployed, due to their self-sufficient, lightweight and portable nature. In this paper, we propose a novel distributed platform using containers for simple execution and evaluation of research applications on the data owner's infrastructure, bringing the algorithms to the data. This approach avoids the cumbersome transfer of large datasets and can help circumventing problems linked to non-shareable data by providing a sandboxed execution environment with read-only access to the data. At no point the data leave the data owner's site, giving researchers access to their evaluation results, only, and not the data themselves. The presented proof-of-concept confirms the feasibility of a distributed container-based evaluation platform for large and/or sensitive data. This has several advantages, including execution of code instead of submission of result files and availability of otherwise inaccessible data. The container architecture allows for minimal computational overhead, no software dependency management on the infrastructure, distributed runtime environment and isolation of processes from the underlying host system. A version addressing various identified architectural and security-related challenges has the potential to be deployed in a production setting and therefore allows researchers to gain insights from previously inaccessible data. One goal is to target hospitals with increasingly strong local infrastructure for storage and computation, needed for artificial intelligence based decision support (genetics and imaging).

show abstract

Design considerations for workflow management systems use in production genomics research and the clinic

Ahmed

Allen

Bhat

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: The changing landscape of genomics research and clinical practice has created a need for computational pipelines capable of efficiently orchestrating complex analysis stages while handling large volumes of data across heterogeneous computational environments. Workflow Management Systems (WfMSs) are the software components employed to fill this gap. Results: This work provides an approach and systematic evaluation of key features of popular bioinformatics WfMSs in use today: Nextflow, CWL, and WDL and some of their executors, along with Swift/T, a workflow manager commonly used in high-scale physics applications. We employed two use cases: a variant-calling genomic pipeline and a scalability-testing framework, where both were run locally, on an HPC cluster, and in the cloud. This allowed for evaluation of those four WfMSs in terms of language expressiveness, modularity, scalability, robustness, reproducibility, interoperability, ease of development, along with adoption and usage in research labs and healthcare settings. This article is trying to answer, "which WfMS should be chosen for a given bioinformatics application regardless of analysis type?". Conclusions: The choice of a given WfMS is a function of both its intrinsic language and engine features. Within bioinformatics, where analysts are a mix of dry and wet lab scientists, the choice is also governed by collaborations and adoption within large consortia and technical support provided by the WfMS team/community. As the community and its needs continue to evolve along with computational infrastructure, WfMSs will also evolve, especially those with permissive licenses that allow commercial use. In much the same way as the dataflow paradigm and containerization are now well understood to be very useful in bioinformatics applications, we will continue to see innovations of tools and utilities for other purposes, like big data technologies, interoperability, and provenance.

show abstract

The impact of Docker containers on the performance of genomic pipelines

Cited by 6 publications

References 3 publications

Automating Installation of the Integrating Biology and the Bedside (i2b2) Platform

Automating Installation of the Integrating Biology and the Bedside (i2b2) Platform

Distributed Container-Based Evaluation Platform for Private/Large Datasets

Design considerations for workflow management systems use in production genomics research and the clinic

Contact Info

Product

Resources

About