A Tale of Two Systems: Using Containers to Deploy HPC Applications on Supercomputers and Clouds

Younge, Andrew J; Pedretti, Kevin; Grant, Ryan E.; Brightwell, Ron

doi:10.1109/cloudcom.2017.40

Cited by 61 publications

(31 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Docker is found to have significant overhead for multi-node MPI applications (Younge et al 2017;Zhang et al 2017); Singularity achieves better performance by directly using the MPI installed on the host machine, but requires compatibility between the MPI libraries inside and outside of the container. These issues will be addressed in future work.…”

Section: Discussionmentioning

confidence: 99%

Enabling Immediate Access to Earth Science Models through Cloud Computing: Application to the GEOS-Chem Model

Zhuang

Jacob

Gaya

et al. 2019

Bulletin of the American Meteorological Society

View full text Add to dashboard Cite

Cloud computing platforms can provide fast and easy access to complex Earth science models and large datasets. This article presents a mature capability for running the GEOS-Chem global 3D model of atmospheric chemistry on the Amazon Web Services (AWS) cloud. GEOS-Chem users at any experience level can get immediate access to the latest, standard version of the model in a preconfigured software environment with all needed meteorological and other input data, and they can analyze model output data easily within the cloud using Python tools in Jupyter notebooks. Users with no prior knowledge of cloud computing are provided with easy-to-follow, step-by-step instructions. They can learn how to complete a demo project in less than one hour, and from there they can configure and submit their own simulations. The cloud is particularly attractive for beginning and occasional users who otherwise may need to spend substantial time configuring a local computing environment. Heavy users with their own local clusters can also benefit from the cloud to access the latest standard model and datasets, share simulation configurations and results, benchmark local simulations, and respond to surges in computing demand. Software containers allow GEOS-Chem and its software environment to be moved smoothly between cloud platforms and local clusters, so that the exact same simulation can be reproduced everywhere. Because the software requirements and workflows tend to be similar across Earth science models, the work presented here provides general guidance for porting models to cloud computing platforms in a user-accessible way.

show abstract

Section: Discussionmentioning

confidence: 99%

Enabling Immediate Access to Earth Science Models through Cloud Computing: Application to the GEOS-Chem Model

Zhuang

Jacob

Gaya

et al. 2019

Bulletin of the American Meteorological Society

View full text Add to dashboard Cite

show abstract

“…On the native non-container environment, in contrast, even if the user installs exactly the same software versions with Spack, there is still chance for operating-system-specific errors to occur, especially for legacy software modules that are not extensively tested on multiple platforms. Despite some pioneering examples of using containers within multi-node MPI environments (Younge et al, 2017;Zhang et al, 2017), such usage can still be challenging in practice, as stated in the Singularity container documentation (https://sylabs.io/guides/3.3/user-guide/mpi.html) that: "the MPI in the container must be compatible with the version of MPI available on the host" and "the configuration of the MPI implementation in the container must be configured for optimal use of the hardware if performance is critical". In this work, we use the native system without containers.…”

Section: Appendix B Approaches To Install Hpc Software Libraries On mentioning

confidence: 99%

Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

Zhuang¹,

Jacob²,

Lin³

et al. 2020

Preprint

View full text Add to dashboard Cite

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input datasets. However, slow inter-node communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services (AWS) cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface (MPI) library configuration and inter-node communication protocol are critical to this success. Application to the GEOS-Chem model of atmospheric chemistry at global 50 km horizontal resolution shows efficient scaling up to at least 1152 cores, with performance and cost comparable to the NASA Pleiades supercomputing cluster.

show abstract

“…Figure 2 shows the details of a possible solution for this example. However, if c 3 was assigned to n 4 , the remaining resources of n 3 would be Rem(n 3 ) = ⟨2, 1024⟩, which are enough to be deployed for c 8 . However, if c 3 was assigned to n 4 , the remaining resources of n 3 would be Rem(n 3 ) = ⟨2, 1024⟩, which are enough to be deployed for c 8 .…”

Section: Container Scheduling Problem Formulationmentioning

confidence: 99%

“…It can be seen that container c 8 is not assigned to any node because none of them has enough remaining resources to allocate this container. However, if c 3 was assigned to n 4 , the remaining resources of n 3 would be Rem(n 3 ) = ⟨2, 1024⟩, which are enough to be deployed for c 8 . This example demonstrates that not every valid solution is optimal.…”

Section: Container Scheduling Problem Formulationmentioning

confidence: 99%

Optimizing scheduling decisions of container management tool using many‐objective genetic algorithm

Imdoukh

Ahmad

Alfailakawi

2019

Concurrency and Computation

View full text Add to dashboard Cite

Container virtualization is an emerging technology in cloud computing mainly due to its portability and lightweight features. Scheduling is a key task, performed by container management tool, which indirectly affects the characteristics of distributed software system in terms of availability, realizability, scalability, resources utilization, as well as power consumption. However, current schedulers only focus on some of the aforementioned aspects but not all. In this paper, a Many-Objective Genetic Algorithm Scheduler (MOGAS) is proposed to handle all such objectives to realize solutions with better characteristics. The proposed scheduler is compared with the Ant Colony Optimization (ACO)-based scheduler. Based on the proposed objective functions, simulation results show that MOGAS is better than the ACO scheduler in equally distributing tasks by 50%, assigning unique set of tasks per node by 40%, and reducing power consumption by 7%, on average.

show abstract

A Tale of Two Systems: Using Containers to Deploy HPC Applications on Supercomputers and Clouds

Cited by 61 publications

References 17 publications

Enabling Immediate Access to Earth Science Models through Cloud Computing: Application to the GEOS-Chem Model

Enabling Immediate Access to Earth Science Models through Cloud Computing: Application to the GEOS-Chem Model

Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

Optimizing scheduling decisions of container management tool using many‐objective genetic algorithm

Contact Info

Product

Resources

About