Amazon Elastic Compute Cloud (EC2) versus In-House HPC Platform: A Cost Analysis

Emeras, Joseph; Varrette, Sébastien; Plugaru, Valentin; Bouvry, Pascal

doi:10.1109/tcc.2016.2628371

Cited by 38 publications

(19 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…DolphinNext readily integrates with Amazon AWS and Google GCP where, a new, dedicated computer cluster can easily be set up within Dolphin-Next with Nextflow's Amazon and Google cloud support. On AWS, necessary input files can be entered from a shared file storage EFS, EBS, or s3, and output files can also be written on s3 or other mounted drives [31][32][33]. On GCP, the input files can be selected from a Google bucket and the output files are exported to another Google bucket.…”

Section: Methodsmentioning

confidence: 99%

DolphinNext: a distributed data processing platform for high throughput genomics

et al. 2020

View full text Add to dashboard Cite

Background: The emergence of high throughput technologies that produce vast amounts of genomic data, such as next-generation sequencing (NGS) is transforming biological research. The dramatic increase in the volume of data, the variety and continuous change of data processing tools, algorithms and databases make analysis the main bottleneck for scientific discovery. The processing of high throughput datasets typically involves many different computational programs, each of which performs a specific step in a pipeline. Given the wide range of applications and organizational infrastructures, there is a great need for highly parallel, flexible, portable, and reproducible data processing frameworks. Several platforms currently exist for the design and execution of complex pipelines. Unfortunately, current platforms lack the necessary combination of parallelism, portability, flexibility and/or reproducibility that are required by the current research environment. To address these shortcomings, workflow frameworks that provide a platform to develop and share portable pipelines have recently arisen. We complement these new platforms by providing a graphical user interface to create, maintain, and execute complex pipelines. Such a platform will simplify robust and reproducible workflow creation for non-technical users as well as provide a robust platform to maintain pipelines for large organizations. Results: To simplify development, maintenance, and execution of complex pipelines we created DolphinNext. DolphinNext facilitates building and deployment of complex pipelines using a modular approach implemented in a graphical interface that relies on the powerful Nextflow workflow framework by providing 1. A drag and drop user interface that visualizes pipelines and allows users to create pipelines without familiarity in underlying programming languages. 2. Modules to execute and monitor pipelines in distributed computing environments such as highperformance clusters and/or cloud 3. Reproducible pipelines with version tracking and stand-alone versions that can be run independently. 4. Modular process design with process revisioning support to increase reusability and pipeline development efficiency. 5. Pipeline sharing with GitHub and automated testing 6. Extensive reports with Rmarkdown and shiny support for interactive data visualization and analysis. Conclusion: DolphinNext is a flexible, intuitive, web-based data processing and analysis platform that enables creating, deploying, sharing, and executing complex Nextflow pipelines with extensive revisioning and interactive reporting to enhance reproducible results.

show abstract

Section: Methodsmentioning

confidence: 99%

DolphinNext: a distributed data processing platform for high throughput genomics

et al. 2020

View full text Add to dashboard Cite

show abstract

“…DolphinNext readily integrates with Amazon AWS where, a new, dedicated computer cluster can easily be set up within DolphinNext with Nextflow's Amazon cloud support. On AWS, necessary input files can be entered from a shared file storage EFS, EBS, or s3, and output files can also be written on s3 or other mounted drives (36)(37)(38).…”

Section: Methodsmentioning

confidence: 99%

DolphinNext: A graphical user interface for creating, deploying and executing Nextflow pipelines

Yukselen

Turkyilmaz

Öztürk

et al. 2019

Preprint

View full text Add to dashboard Cite

Several platforms currently exist for the design and execution of complex pipelines (e.g. Galaxy, GenePattern, GeneProf). Unfortunately, these platforms lack the necessary combination of parallelism, portability, flexibility and/or reproducibility that are required by the current research environment. To address these shortcomings, Nextflow was implemented to simplify portable, scalable, and reproducible scientific pipelines using containers. We have used Nextflow capabilities as leverage and developed a user interface, DolphinNext, for creating, deploying, and executing complex Nextflow pipelines for high throughput data processing. The guiding principle of DolphinNext is to facilitate the building and deployment of complex pipelines using a modular approach implemented in a graphical interface. DolphinNext provides: 1. A drag and drop user interface that abstracts pipelines and allows users to create pipelines without familiarity in underlying programming languages. 2. A user interface to monitor pipeline execution that allows the re-initiation of pipelines at intermediate steps 3. Reproducible pipelines with version tracking and stand-alone versions that can be run independently. 4. Seamless portability to distributed computational environments such as high-performance clusters or cloud computing environments.

show abstract

“…• Understanding the Performance and Potential of Cloud Computing for Scientific Applications [25] Problema: As aplicações científicas geralmente exigem recursos significativos, no entanto, nem todos os • Amazon Elastic Compute Cloud (EC2) versus In-House HPC Platform: A Cost Analysis [26] Problema: Embora os centros de computação de alto desempenho (HPC) evoluam continuamente para fornecer mais poder de computação a seus usuários, observase um desejo de convergência entre plataformas de computação em nuvem (CC) e computação de alto desempenho (HPC). Excluindo-se o ponto de vista do desempenho, em que muitos estudos destacam uma sobrecarga induzida pela camada de virtualização no núcleo dos middlewares em nuvem ao executar uma carga de trabalho HPC, a relação custo-benefício real costuma ser deixada de lado com o desejo de que as instâncias oferecidas pelos provedores de nuvem sejam competitivas do ponto de vista dos custos.…”

Section: Relatório Da Rslunclassified

“…No seu trabalho, [26] (T11) propôs analisar os elementos que compõe o custo total de propriedade (TCO) de uma estrutura interna de HPC, operando desde 2007 e, de posse das informações, comparar os custos necessários para executar a mesma carga de trabalho em uma estrutura de nuvem pública. Os resultados obtidos mostram que a migração de cargas de trabalho HPC para nuvem não é apenas um problema de adaptabilidade do desempenho da nuvem às necessidades das aplicações HPC, mas também um problema ao determinar corretamente quais tipos de trabalho são bons candidatos a serem executados na nuvem para evitar sobrecarga de custos.…”

Section: Hpcunclassified

Cloud infrastructure for HPC investment analysis

Santos

Cavalheiro

2020

RITA

View full text Add to dashboard Cite

With the consolidation of cloud computing technology, there is a growing interest in exploring it to support High Performance Computing (HPC). However, migrating such applications to public or private cloud environments brings some challenges, in particular, the cost in financing the migration process. In this paper, a literature review is presented with selected papers about analyzing cloud infrastructure investments. In particular, the selected papers analyse how investments impact applications. For discussion of related works, conditions for running HPC applications in the cloud are characterized.

show abstract

Amazon Elastic Compute Cloud (EC2) versus In-House HPC Platform: A Cost Analysis

Cited by 38 publications

References 21 publications

DolphinNext: a distributed data processing platform for high throughput genomics

DolphinNext: a distributed data processing platform for high throughput genomics

DolphinNext: A graphical user interface for creating, deploying and executing Nextflow pipelines

Cloud infrastructure for HPC investment analysis

Contact Info

Product

Resources

About