Experiences building Globus Genomics: a next‐generation sequencing analysis service using Galaxy, Globus, and Amazon Web Services

Madduri, Ravi; Sulakhe, Dinanath; Łaciński, Łukasz; Liu, Bo; Rodríguez, Álex; Chard, Kyle; Dave, Utpal J.; Foster, Ian

doi:10.1002/cpe.3274

Cited by 67 publications

(50 citation statements)

References 26 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We aim to leverage billing capabilities we have developed for the Globus Genomics [33] service for this purpose. The Globus Genomics model charges users for the cumulative costs associated with each analysis job as well as a fixed subscription for using the service.…”

Section: Pricing Strategiesmentioning

confidence: 99%

Scalable pCT Image Reconstruction Delivered as a Cloud Service

Chard

Madduri

Karonis

et al. 2018

IEEE Trans. Cloud Comput.

Self Cite

View full text Add to dashboard Cite

We describe a cloud-based medical image reconstruction service designed to meet a real-time and daily demand to reconstruct thousands of images from proton cancer treatment facilities worldwide. Rapid reconstruction of a three-dimensional Proton Computed Tomography (pCT) image can require the transfer of 100GB of data and use of approximately 120 GPU-enabled compute nodes. The nature of proton therapy means that demand for such a service is sporadic and comes from potentially hundreds of clients worldwide. We thus explore the use of a commercial cloud as a scalable and cost-efficient platform for pCT reconstruction. To address the high performance requirements of this application we leverage Amazon Web Services' GPU-enabled cluster resources that are provisioned with high performance networks between nodes. To support episodic demand, we develop an on-demand multi-user provisioning service that can dynamically provision and resize clusters based on image reconstruction requirements, priorities, and wait times. We compare the performance of our pCT reconstruction service running on commercial cloud resources with that of the same application on dedicated local high performance computing resources. We show that we can achieve scalable and on-demand reconstruction of large scale pCT images for simultaneous multi-client requests, processing images in less than ten minutes for less than $10 per image.

show abstract

Section: Pricing Strategiesmentioning

confidence: 99%

Scalable pCT Image Reconstruction Delivered as a Cloud Service

Chard

Madduri

Karonis

et al. 2018

IEEE Trans. Cloud Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

“…To address these challenges, Globus Genomics [58,59] was developed at the Computation Institute (CI), a joint institute between the University of Chicago and Argonne National Laboratory. Globus Genomics is a cloud-based integrated solution for NGS data analysis.…”

Section: Bioinformatics Platformsmentioning

confidence: 99%

“…It extends the existing Galaxy workflow system by adding data management capabilities for transferring large quantities of data efficiently and reliably (via Globus Transfer), domain-specific analyses tools preconfigured for immediate use by researchers (via user-specific tools integration), automatic deployment on cloud for on-demand resource allocation and pay-as-you-go pricing (via Globus Provision), and a cloud provisioning tool for auto-scaling (via HTCondor scheduler). Genome sequencing is notoriously data-intensive, and Globus Transfer [59] is designed for fast and secure movement of large amounts of data. Setting up a production instance of Galaxy is a nontrivial task that involves a number of manual installation and configuration steps for both the platform and any dependent software packages-steps that can be both error-prone and time-consuming.…”

Section: Bioinformatics Platformsmentioning

confidence: 99%

Cloud Computing for Next-Generation Sequencing Data Analysis

Zhao¹,

Watrous²,

Zhang³

et al. 2017

Cloud Computing - Architecture and Applications

View full text Add to dashboard Cite

High-throughput next-generation sequencing (NGS) technologies have evolved rapidly and are reshaping the scope of genomics research. The substantial decrease in the cost of NGS techniques in the past decade has led to its rapid adoption in biological research and drug development. Genomics studies of large populations are producing a huge amount of data, giving rise to computational issues around the storage, transfer, and analysis of the data. Fortunately, cloud computing has recently emerged as a viable option to quickly and easily acquire the computational resources for large-scale NGS data analyses. Some cloud-based applications and resources have been developed specifically to address the computational challenges of working with very large volumes of data generated by NGS technology. In this chapter, we will review some cloud-based systems and solutions for NGS data analysis, discuss the practical hurdles and limitations in cloud computing, including data transfer and security, and share the lessons we learned from the implementation of Rainbow, a cloud-based tool for large-scale genome sequencing data analysis.

show abstract

“…Большая часть из них является коммерческими продуктами, среди них наиболее известными и широко используемыми являются BaseSpace (Illumina, Inc.) [13], CLCBio [14], Lasergene (DNASTAR, Inc.) [15], Geneious [16]. Среди свободных и гибридных (частично коммерческих, частично свободных) популярными являются программные комплексы Galaxy [17,18], Globus Genomics [19], PATRIC [20], UGENE [21,22]. Они обладают широчайшими возможностями, большинство из этих программных пакетов могут быть установлены на локальный компьютер или сервер, они работают под разными операционными системами.…”

Section: программное обеспечениеunclassified

Big Data in Bioinformatics

Nazipova¹

2017

Math.Biol.Bioinf.

View full text Add to dashboard Cite

Аннотация. Секвенирование человеческого генома началось в 1994 году. Понадобилось 10 лет работы многих научных коллективов для того, чтобы получить черновую последовательность ДНК человека. Современные технологии секвенирования позволяют получать геном конкретного человека за несколько дней. Обсуждаются успехи современной биоинформатики, связанные с появлением высокопроизводительных платформ секвенирования, которые не только способствовали расширению возможностей различных направлений биологии и других смежных наук, но и породили феномен больших данных. Обосновывается необходимость разработки новых технологий и методов для организации хранения, управления, анализа и визуализации больших данных. Современная биоинформатика столкнулась не только с проблемой больших данных, но и с огромным разнообразием методов обработки и представления, одновременным существованием различных программных средств и форматов данных. Обсуждаются пути решения возникших проблем, в частности путем использования наработок работы с большими данными из других областей современной жизни, таких как сетевой анализ и анализ деловых данных. Новые системы управления базами данных, отличные от реляционных, помогут решить проблему хранения больших данных и обеспечения приемлемого времени выполнения поисковых запросов. Новые технологии программирования, такие, как обобщенное программирование и визуальное программирование призваны решить проблему разнообразия форматов геномных данных и обеспечить возможность оперативного создания собственных скриптов для обработки данных.

show abstract

Experiences building Globus Genomics: a next‐generation sequencing analysis service using Galaxy, Globus, and Amazon Web Services

Cited by 67 publications

References 26 publications

Scalable pCT Image Reconstruction Delivered as a Cloud Service

Scalable pCT Image Reconstruction Delivered as a Cloud Service

Cloud Computing for Next-Generation Sequencing Data Analysis

Big Data in Bioinformatics

Contact Info

Product

Resources

About