Exploring Cloud Computing for Large-Scale Scientific Applications

Lin, Guang; Han, Binh; Yin, Jianwei; Gorton, Ian

doi:10.1109/services.2013.13

Cited by 6 publications

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several works have addressed the opportunities of shifting scientific workflows from traditional HPC and HTC infrastructures to CC platforms. In particular, authors have focused on exploring data-intensive workflows, since they are the most tightly related to conventional BD applications in terms of data volumes [55,27]. Experimentation with well known workflows, like Montage, shows that running costs could be significantly decreased with CC infrastructures, but performance would suffer from virtualization and latency overheads [15,22,3].…”

Section: Cloud Computingmentioning

confidence: 99%

Applying big data paradigms to a large scale scientific workflow: Lessons learned and future directions

Caíno‐Lores

Lapin

Carretero

et al. 2020

Future Generation Computer Systems

View full text Add to dashboard Cite

Section: Cloud Computingmentioning

confidence: 99%

Applying big data paradigms to a large scale scientific workflow: Lessons learned and future directions

Caíno‐Lores

Lapin

Carretero

et al. 2020

Future Generation Computer Systems

View full text Add to dashboard Cite

“…Several works have addressed the opportunities of shifting scientific workflows from traditional HPC and HTC infrastructures to BDA computing infrastructures such as clouds. In particular, authors have focused on exploring data-intensive workflows, since they are the most tightly related to conventional BDA applications in terms of data volumes [126], [127]. Experimentation with well known workflows shows that running costs could be significantly decreased with BDA infrastructures, but performance would suffer from virtualization and latency overheads [128]- [130].…”

Section: ) Infrastructure: Distributed Storage and Cloud Computingmentioning

confidence: 99%

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

et al. 2019

View full text Add to dashboard Cite

Convergence between high-performance computing (HPC) and big data analytics (BDA) is currently an established research area that has spawned new opportunities for unifying the platform layer and data abstractions in these ecosystems. This work presents an architectural model that enables the interoperability of established BDA and HPC execution models, reflecting the key design features that interest both the HPC and BDA communities, and including an abstract data collection and operational model that generates a unified interface for hybrid applications. This architecture can be implemented in different ways depending on the process-and data-centric platforms of choice and the mechanisms put in place to effectively meet the requirements of the architecture. The Spark-DIY platform is introduced in the paper as a prototype implementation of the architecture proposed. It preserves the interfaces and execution environment of the popular BDA platform Apache Spark, making it compatible with any Spark-based application and tool, while providing efficient communication and kernel execution via DIY, a powerful communication pattern library built on top of MPI. Later, Spark-DIY is analyzed in terms of performance by building a representative use case from the hydrogeology domain, EnKF-HGS. This application is a clear example of how current HPC simulations are evolving toward hybrid HPC-BDA applications, integrating HPC simulations within a BDA environment. INDEX TERMS Big data analytics, high performance computing, spark, DIY, MPI.

show abstract

CCMA—cloud critical metric assessment framework for scientific computing

Ravindhren¹,

Ravimaran

2017

Cluster Comput

View full text Add to dashboard Cite

Exploring Cloud Computing for Large-Scale Scientific Applications

Cited by 6 publications

References 12 publications

Applying big data paradigms to a large scale scientific workflow: Lessons learned and future directions

Applying big data paradigms to a large scale scientific workflow: Lessons learned and future directions

Toward High-Performance Computing and Big Data Analytics Convergence: The Case of Spark-DIY

CCMA—cloud critical metric assessment framework for scientific computing

Contact Info

Product

Resources

About