Exposing HPC and sequential applications as services through the development and deployment of a SaaS cloud

Church, Philip C.; Gościński, Andrzej; Lefèvre, Christophe

doi:10.1016/j.future.2014.10.001

Cited by 18 publications

(14 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Building a multinode HPC cluster environment on the clouds used to be difficult, prompting developments of “HPC‐as‐a‐Service” (Baun et al, 2011; Church et al, 2015; Huang, 2014; Section 2.3 of Netto et al, 2017; Wong & Goscinski, 2013), where scientists can access a preconfigured HPC environment, often through a graphical web portal (Calegari et al, 2019), with no need to understand the underlying infrastructure. However, such black‐box service has several drawbacks for research computing: (1) The available software libraries and applications are determined by the HPC service provider and are not easy to extend to custom research code and (2) continuous maintenance by the service provider is required to keep software and hardware up to date.…”

Section: Hpc Workflow On the Aws Cloudmentioning

confidence: 99%

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Zhuang

Jacob

Lin

et al. 2020

J Adv Model Earth Syst

View full text Add to dashboard Cite

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input data sets. However, slow internode communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface library configuration and internode communication protocol are critical to this success. Application to the Goddard Earth Observing System (GEOS)-Chem global 3-D chemical transport model at 50-km horizontal resolution shows efficient scaling up to at least 1,152 cores, with performance and cost comparable to the National Aeronautics and Space Administration Pleiades supercomputing cluster.Plain Language Summary Earth science model simulations are computationally expensive, typically requiring the use of high-end supercomputing clusters that are managed by universities or national laboratories. Commercial cloud computing offers an alternative. However, past work found that cloud computing platforms were not efficient for large-scale simulations on over 100 CPU cores, because the network communication performance on the cloud was slow compared to local clusters. Here we show that recent advances in the cloud network performance enable efficient model simulations with over a thousand cores, and cloud platforms can now serve as a viable alternative to local clusters for simulations at large scale. Computing on the cloud has extensive advantages, such as providing immediate access to fully configured model code and large data sets for any users, allowing full reproducibility of model simulation results, offering quick access to novel hardware that might not be available on local clusters, and being able to scale to virtually unlimited amounts of compute and storage resources. Those benefits will help advance Earth science modeling research.

show abstract

Section: Hpc Workflow On the Aws Cloudmentioning

confidence: 99%

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Zhuang

Jacob

Lin

et al. 2020

J Adv Model Earth Syst

View full text Add to dashboard Cite

show abstract

“…Building a multi-node HPC cluster environment on the clouds used to be difficult, prompting developments of "HPC-as-a-Service" (Baun et al, 2011;Wong and Goscinski, 2013;Huang, 2014;Church et al, 2015;Section 2.3 of Netto et al, 2017), where scientists can access a preconfigured HPC environment, often through a graphical web portal (Calegari et al, 2019), with no need to understand the underlying infrastructure. However, such black-box service has several drawbacks for research computing: (1) the available software libraries and applications are determined by the HPC service provider, and are not easy to extend to custom research code; (2) continuous maintenance by the service provider is required to keep software and hardware up-todate.…”

Section: Multi-node Workflowmentioning

confidence: 99%

Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

Zhuang¹,

Jacob²,

Lin³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Having this motivation, they started to develop a framework to translate HPC applications into services. Over the years, the same group [31] presented novel use cases of their framework in the area of genomics. Church and Goscinki [33] also presented a survey on technologies available to help researchers working with mammalian genomics run their experiments.…”

Section: Execution Steeringmentioning

confidence: 99%

“…Management [18,31,33,115,117] Related to proper mapping of multiple activities, possibly including dependencies. Frameworks that have knowledge of cloud pricing models can reduce time and costs for deploying and exposing HPC applications in the cloud.…”

Section: Workflowmentioning

confidence: 99%

HPC Cloud for Scientific and Business Applications

et al. 2018

View full text Add to dashboard Cite

High Performance Computing (HPC) clouds are becoming an alternative to on-premise clusters for executing scientific applications and business analytics services. Most research efforts in HPC cloud aim to understand the cost-benefit of moving resource-intensive applications from on-premise environments to public cloud platforms. Industry trends show hybrid environments are the natural path to get the best of the on-premise and cloud resources-steady (and sensitive) workloads can run on on-premise resources and peak demand can leverage remote resources in a pay-as-you-go manner. Nevertheless, there are plenty of questions to be answered in HPC cloud, which range from how to extract the best performance of an unknown underlying platform to what services are essential to make its usage easier. Moreover, the discussion on the right pricing and contractual models to fit small and large users is relevant for the sustainability of HPC clouds. This paper brings a survey and taxonomy of efforts in HPC cloud and a vision on what we believe is ahead of us, including a set of research challenges that, once tackled, can help advance businesses and scientific discoveries. This becomes particularly relevant due to the fast increasing wave of new HPC applications coming from big data and artificial intelligence. users have no visibility or concerns on costs of running jobs. However, large clusters do incur expenses and, when not properly managed, can generate resource wastage and poor quality of service.Motivated by the different utilization levels of clusters around the globe and by the need to run even larger parallel programs, in the early 2000s, Grid Computing became relevant for the HPC community. Grids offer users access to powerful resources managed by autonomous administrative domains [50,51]. The notion of monetary costs for running applications was soft, favoring a more collaborative model of resource sharing. Therefore, quality of service was not strict in Grids, having users relying on best-effort policies to run applications.In the late 2000s, cloud computing [8,26,91] was quickly increasing its maturity level and popularity, and studies started to emerge on the viability of executing HPC applications on remote cloud resources. These applications, which consume more resources than traditional cloud applications and usually are executed in batches rather than 24x7 services, range from parallel applications written in Message Passing Interface (MPI) [58,59] to the newest big data [11,14,39,101] and artificial intelligence applications-the latter mostly relying on deep learning [34,80]. Cloud then came up as an evolution of a series of technologies, mainly on virtualization and computer networks, which facilitated both workload management and interaction with remote resources respectively. Apart from software and hardware, cloud offers a business model where users pay for resources on demand. Compared to traditional HPC environments, in clouds users can quickly adjust their resource pools, via a mechanism known as elast...

show abstract

Exposing HPC and sequential applications as services through the development and deployment of a SaaS cloud

Cited by 18 publications

References 19 publications

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

HPC Cloud for Scientific and Business Applications

Contact Info

Product

Resources

About