Enabling EASEY Deployment of Containerized Applications for Future HPC Systems

Höb, Maximilian; Kranzlmüller, Dieter

doi:10.1007/978-3-030-50371-0_15

Cited by 12 publications

(11 citation statements)

References 12 publications

(12 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, very few approaches to automating its deployment can be found in the literature. EASEY introduces a framework to enable container applications based on Docker to be transformed automatically to Charliecloud containers and executed on HPC systems via an integrated management system (Höb and Kranzlmüller, 2020). The architecture of the EASEY system is integrated as two building bricks in the layered HPC architecture.…”

Section: State Of the Artmentioning

confidence: 99%

Automatizing the creation of specialized high-performance computing containers

Ejarque

Badía

2023

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

With Exascale computing already here, supercomputers are systems every time larger, more complex, and heterogeneous. While expert system administrators can install and deploy applications in the systems correctly, this is something that general users can not usually do. The eFlows4HPC project aims to provide methodologies and tools to enable the use and reuse of application workflows. One of the aspects that the project focuses on is simplifying the application deployment in large and complex systems. The approach uses containers, not generic ones, but containers tailored for each target High-Performance Computing (HPC) system. This paper presents the Container Image Creation service developed in the framework of the project and experimentation based on project applications. We compare the performance of the specialized containers against generic containers and against a native installation. The results show that in almost all cases, the specialized containers outperform the generic ones (up to 2× faster), and in all cases, the performance is the same as with the native installation.

show abstract

Section: State Of the Artmentioning

confidence: 99%

Automatizing the creation of specialized high-performance computing containers

Ejarque

Badía

2023

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

show abstract

“…In [1] we introduced a deployment framework, which enables scientist to easily convert a docker container to a charliecloud container while including specific target system dependencies and libraries. This is especially important since more and more HPC-unexperienced scientist want and need to use supercomputing facilities, because their own smaller system are no longer capable of their own computation.…”

Section: Easey Backgroundmentioning

confidence: 99%

“…The initial architecture of the EASEY framework as presented in [1] was updated with a new layer, the EASEY ANALYZER and is displayed in Figure 1 on page 3. Together with the existing building bricks it was integrated in the layered HPC architecture.…”

Section: Maximilian Höbmentioning

confidence: 99%

“…We base our approach on the EASEY (Enable exASclae for EverYone in [1]) framework, which can automatically deploy optimized container computations with negligible overhead. Todays containers are natively not able to automatically use all given hardware at best, since the encapsulated application varies on computing, memory or communication demands.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Performance Characterization of Containerized HPC Workloads

Hob

2023

Proceedings of International Symposium on Grids &Amp; Clouds (ISGC) 2023 in Conjunction With HEPiX Spring 2023 Workshop — PoS(I

View full text Add to dashboard Cite

In this paper, we address the complex challenges that arise within existing high performance computing (HPC) frameworks as we approach the exascale era. On one side, highly optimized, heterogeneous hardware systems coexist with HPC-unexperienced scientists with increasing demand for compute and data capacity. We propose containerization as a key concept to shift the focus back to the actual domain science, enabling an efficient usage of the compute systems and removing incompatible dependencies, unsupported subprograms or compilation challenges. To this end, we provide a methodology to determine, analyze and evaluate characteristic parameters of containerized HPC applications to fingerprint the overall performance of arbitrary containerized applications. The methodology comprises the performance parameter definition and selection, a measurement method to minimize overhead, and a fingerprinting algorithm to enable characteristics comparison and mapping between application and target system. We apply the methodology to benchmark applications to demonstrate its capability.

show abstract

“…Recently, work has been developed that explores the future difficulties that researchers will face when scaling up their experiments on HPC clusters. In [37] the difficulties involved in moving workflows from CPU-based to GPU-based systems are addressed. Authors propose initial steps towards a framework for the automatic deployment of optimized containers in such systems.…”

Section: Related Workmentioning

confidence: 99%

A Container-Based Workflow for Distributed Training of Deep Learning Algorithms in HPC Clusters

González-Abad¹,

García²,

Козлов³

2022

Preprint

View full text Add to dashboard Cite

Deep learning has been postulated as a solution for numerous problems in different branches of science. Given the resource-intensive nature of these models, they often need to be executed on specialized hardware such graphical processing units (GPUs) in a distributed manner. In the academic field, researchers get access to this kind of resources through High Performance Computing (HPC) clusters. This kind of infrastructures make the training of these models difficult due to their multi-user nature and limited user permission. In addition, different HPC clusters may possess different peculiarities that can entangle the research cycle (e.g., libraries dependencies). In this paper we develop a workflow and methodology for the distributed training of deep learning models in HPC clusters which provides researchers with a series of novel advantages. It relies on udocker as containerization tool and on Horovod as library for the distribution of the models across multiple GPUs. udocker does not need any special permission, allowing researchers to run the entire workflow without relying on any administrator. Horovod ensures the efficient distribution of the training independently of the deep learning framework used. Additionally, due to containerization

show abstract

Enabling EASEY Deployment of Containerized Applications for Future HPC Systems

Cited by 12 publications

References 12 publications

Automatizing the creation of specialized high-performance computing containers

Automatizing the creation of specialized high-performance computing containers

Performance Characterization of Containerized HPC Workloads

A Container-Based Workflow for Distributed Training of Deep Learning Algorithms in HPC Clusters

Contact Info

Product

Resources

About