Strategies for Fault-Tolerant Tightly-Coupled HPC Workloads Running on Low-Budget Spot Cloud Infrastructures

Munhoz, Vanderlei; Castro, Márcio; Mendizabal, Odorico Machado

doi:10.1109/sbac-pad55451.2022.00037

Cited by 8 publications

(10 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…23 We integrated into HPC@Cloud existing fault tolerance strategies to address the unreliability of spot instances. 6 These strategies are built upon existing technologies such as BLCR and the innovative ULFM, which is currently under development by the MPI Forum. Although numerous studies have explored fault tolerance using BLCR and other well-established technologies, they often do not comprehensively analyze the advantages and disadvantages of utilizing more affordable, less reliable cloud instances compared to high-end options in a public cloud and HPC context.…”

Section: Related Workmentioning

confidence: 99%

“…We integrated into HPC@Cloud existing fault tolerance strategies to address the unreliability of spot instances 6 . These strategies are built upon existing technologies such as BLCR and the innovative ULFM, which is currently under development by the MPI Forum.…”

Section: Related Workmentioning

confidence: 99%

“…Although numerous studies have explored fault tolerance using BLCR and other well‐established technologies, they often do not comprehensively analyze the advantages and disadvantages of utilizing more affordable, less reliable cloud instances compared to high‐end options in a public cloud and HPC context. We also propose and evaluate a new adaptive fault tolerance strategy based on ULFM that does not stop the MPI application when AWS revokes spot instances, thus providing better performance than blocking fault tolerance strategies proposed in Reference 6.…”

Section: Related Workmentioning

confidence: 99%

“…To support spot instances and other unreliable infrastructures, the Job Manager provides a set of fault tolerance strategies to resume execution and restore spot clusters through new spot requests. We integrated into HPC@Cloud two blocking restoration strategies proposed in Reference 6: one based on a system‐level rollback restart mechanism using the BLCR package 25 and other based on the novel ULFM approach for checkpoint‐restart. Moreover, we also introduce an entirely new adaptive restoration strategy based on ULFM that keeps applications running while waiting for new spot instances to be created.…”

Section: The Hpc@cloud Toolkitmentioning

confidence: 99%

“…It features a flexible command line interface that allows researchers to easily specify the required computing resources and configurations when building the HPC infrastructure on public cloud providers. MPI applications can run seamlessly on the provisioned HPC cluster using TCP/IP or InfiniBand (when available) for communications; We integrate into HPC@Cloud all blocking fault tolerance strategies tailored to tightly‐coupled Message Passing Interface (MPI) applications running on AWS spot instances proposed in Reference 6; We propose a new adaptive fault tolerance strategy that does not stop the MPI application when AWS revokes spot instances, thus providing better performance than blocking fault tolerance strategies proposed in Reference 6; We introduce support for Singularity containers, allowing more portability when migrating legacy HPC code to the cloud, and evaluate their performance overheads; We propose an empirical approach for estimating job execution costs on public clouds based on the execution of the target application with a small input problem size and gathered execution metrics, then later used to train a regression model for predicting total execution time; and We provide a practical cost‐efficiency analysis of executing a set of NAS Parallel Benchmarks (NPB) applications 7 and a parallel solver implementation for Laplace's heat diffusion equations based on a popular alternative Finite Difference Method (FDM) technique called Forward‐Time Central Space (FTCS) method 5,8 on two popular public cloud platforms (AWS and Vultr Cloud) across multiple cluster configurations, giving an in‐depth analysis of the advantages, main performance bottlenecks, costs, and pitfalls of using public cloud resources for HPC.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Enabling the execution of HPC applications on public clouds with HPC@Cloud toolkit

Munhoz,

Castro

2023

Concurrency and Computation

View full text Add to dashboard Cite

The advent of cloud computing has made access to computing infrastructure available to millions of users that face resource constraints. In the context of high performance computing (HPC), public cloud resources have emerged as a cost‐effective alternative to expensive on‐premises clusters. However, there are several challenges and limitations in adopting this approach. This paper proposes HPC@Cloud , a provider‐agnostic open‐source software toolkit that facilitates the migration, testing, and execution of HPC applications in public clouds. The toolkit takes advantage of various fault tolerance technologies to enable the use of inexpensive transient cloud infrastructure, commonly known as “spot” instances. Also, it features integration with singularity containers, allowing users to run complex applications on virtual HPC clusters in a portable and reproducible way. Finally, it provides a data‐based empirical approach to estimating cloud infrastructure costs for HPC workloads. The results obtained on two public cloud providers (AWS and Vultr) show that: (i) HPC@Cloud can efficiently build virtual HPC clusters on the cloud; (ii) the new adaptive fault tolerance strategy outperforms other existing strategies based on blocking restoration; (iii) the integration of singularity containers into HPC@Cloud improves the portability of HPC applications to public clouds with negligible performance penalty to the applications; (iv) the proposed cost prediction approach can estimate the cost of running the applications on AWS and Vultr with up to 93% accuracy on average.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: The Hpc@cloud Toolkitmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Enabling the execution of HPC applications on public clouds with HPC@Cloud toolkit

Munhoz,

Castro

2023

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

Improving batch schedulers with node stealing for failed jobs

Du,

Marchal,

Pallez

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

SummaryAfter a machine failure, batch schedulers typically re‐schedule the job that failed with a high priority. This is fair for the failed job but still requires that job to re‐enter the submission queue and to wait for enough resources to become available. The waiting time can be very long when the job is large and the platform highly loaded, as is the case with typical HPC platforms. We propose another strategy: when a job fails, if no platform node is available, we steal one node from another job , and use it to continue the execution of despite the failure. In this work, we give a detailed assessment of this node stealing strategy using traces from the Mira supercomputer at Argonne National Laboratory. The main conclusion is that node stealing improves the utilization of the platform and dramatically reduces the flow of large jobs, at the price of slightly increasing the flow of small jobs.

show abstract