MIT SuperCloud portal workspace: Enabling HPC web application deployment

Prout, Andrew; Arcand, William; Bestor, David; Bergeron, Bill; Byun, Chansup; Gadepally, Vijay; Hubbell, Matthew; Houle, Michael E.; Jones, Michael; Michaleas, Peter; Milechin, Lauren; Mullen, Julie; Rosa, Antonio; Samsi, Siddharth; Reuther, Albert; Kepner, Jeremy

doi:10.1109/hpec.2017.8091097

Cited by 9 publications

(9 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Appendix shows the advantages of Spack compared to other software installation methods. If desired, it is then possible to deploy HPC‐as‐a‐Service for other users on top of such a custom HPC environment, using auxiliary AWS functionalities (Amazon, 2019f, 2019a, 2019c) and multiuser serving frameworks like JupyterHub (Glick & MacHe, 2019; Milligan, 2018; Prout et al, 2017; Sarajlic et al, 2018); this can be useful for classes and workshops where the customization of software environment is not important.…”

Section: Hpc Workflow On the Aws Cloudmentioning

confidence: 99%

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Zhuang

Jacob

Lin

et al. 2020

J Adv Model Earth Syst

View full text Add to dashboard Cite

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input data sets. However, slow internode communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface library configuration and internode communication protocol are critical to this success. Application to the Goddard Earth Observing System (GEOS)-Chem global 3-D chemical transport model at 50-km horizontal resolution shows efficient scaling up to at least 1,152 cores, with performance and cost comparable to the National Aeronautics and Space Administration Pleiades supercomputing cluster.Plain Language Summary Earth science model simulations are computationally expensive, typically requiring the use of high-end supercomputing clusters that are managed by universities or national laboratories. Commercial cloud computing offers an alternative. However, past work found that cloud computing platforms were not efficient for large-scale simulations on over 100 CPU cores, because the network communication performance on the cloud was slow compared to local clusters. Here we show that recent advances in the cloud network performance enable efficient model simulations with over a thousand cores, and cloud platforms can now serve as a viable alternative to local clusters for simulations at large scale. Computing on the cloud has extensive advantages, such as providing immediate access to fully configured model code and large data sets for any users, allowing full reproducibility of model simulation results, offering quick access to novel hardware that might not be available on local clusters, and being able to scale to virtually unlimited amounts of compute and storage resources. Those benefits will help advance Earth science modeling research.

show abstract

Section: Hpc Workflow On the Aws Cloudmentioning

confidence: 99%

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Zhuang

Jacob

Lin

et al. 2020

J Adv Model Earth Syst

View full text Add to dashboard Cite

show abstract

“…The MIT Supercloud enables traditional enterprise computing and cloud computing workloads to be run on an HPC cluster. The software stack, which contains all of the system and application software, resides on every node [26].…”

Section: B High Performance Computing Environmentmentioning

confidence: 99%

“…HPC systems require efficient mechanisms for rapidly identifying available computing resources, allocating those resources to programs, and launching the programs on the allocated resources. The open-source SLURM software [27] provides scalable cluster management, a job scheduling sys-tem, and is independent of programming language or parallel programming models [26].…”

Section: B High Performance Computing Environmentmentioning

confidence: 99%

Active Learning Pipeline for Brain Mapping in a High Performance Computing Environment

Michaleas¹,

Gjesteby²,

Snyder³

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper describes a scalable active learning pipeline prototype for large-scale brain mapping that leverages high performance computing power. It enables high-throughput evaluation of algorithm results, which, after human review, are used for iterative machine learning model training. Image processing and machine learning are performed in a batch layer. Benchmark testing of image processing using pMATLAB shows that a 100× increase in throughput (10,000%) can be achieved while total processing time only increases by 9% on Xeon-G6 CPUs and by 22% on Xeon-E5 CPUs, indicating robust scalability. The images and algorithm results are provided through a serving layer to a browser-based user interface for interactive review. This pipeline has the potential to greatly reduce the manual annotation burden and improve the overall performance of machine learning-based brain mapping.

show abstract

“…A new type of HPC portal exists for web-based HPC applications. For example, the MIT Super-Cloud portal workspace [110] provides authentication, encryption, and access control to enable the secure exposure of web services running on HPC systems. Such a portal acts as a reverse proxy for accessing web applications ran as HPC jobs by end-users.…”

Section: Hpc Portal Listmentioning

confidence: 99%

Untitled

2019

ACM Trans. Web

View full text Add to dashboard Cite

This article addresses web interfaces for High-performance Computing (HPC) simulation software. First, it presents a brief history, starting in the 1990s with Java applets, of web interfaces used for accessing and making best possible use of remote HPC resources. It introduces HPC web-based portal use cases. Then it identifies and discusses the key features, among functional and non-functional requirements, that characterize such portals. A brief state of the art is then presented. The design and development of Bull extreme factory Computing Studio v3 (XCS3) is chosen as a common thread for showing how the identified key features can all be implemented in one software: multi-tenancy, multi-scheduler compatibility, complete control through an HTTP RESTful API, customizable user interface with Responsive Web Design, HPC application template framework, remote visualization, and access through the Authentication, Authorization, and Accounting security framework with the Role-Based Access Control permission model. Non-functional requirements (security, usability, performance, reliability) are discussed, and the article concludes by giving perspective for future work. CCS Concepts: • Human-centered computing → Graphical user interfaces; • Information systems → RESTful web services; • Computing methodologies → Massively parallel and high-performance simulations; • Software and its engineering → Cloud computing;

show abstract

MIT SuperCloud portal workspace: Enabling HPC web application deployment

Cited by 9 publications

References 18 publications

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Active Learning Pipeline for Brain Mapping in a High Performance Computing Environment

Untitled

Contact Info

Product

Resources

About