Application migration in HPC — A driver of the exascale era?

Pickartz, Simon; Lankes, Stefan; Monti, Antonello; Clauss, Carsten; Breitbart, Jens

doi:10.1109/hpcsim.2016.7568352

Cited by 13 publications

(15 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The time to reconnect (black surface) is negligibly small and, hence, not visible since we leverage the pscom on‐demand mechanism. Therefore, the actual reconnect is postponed to the first usage of that connection (based on the authors' previous work)…”

Section: A Protocol Enabling the Migration Of Mpi Processesmentioning

confidence: 99%

“…Abandoning the concept of static schedules, jobs or parts thereof can be migrated during runtime across the cluster for the avoidance of load imbalances. () This is especially interesting for legacy applications, which are not likely to be tuned to every new generation of supercomputers in many cases. Furthermore, migration mechanisms allow for an eased system maintenance by the evacuation of individual nodes .…”

Section: Introductionmentioning

confidence: 99%

“…This work builds upon our research in the field of application migration in HPC() and co‐scheduling techniques enabled by virtualization using VMs. () In doing so, we present the following substantial contributions: a thorough evaluation of our virtualization‐aware Message Passing Interface (MPI) library, a locality‐aware extension with support for dynamically changing topologies, and a prototype co‐scheduler leveraging VM migration. …”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Prospects and challenges of virtual machine migration in HPC

Pickartz

Clauss

Breitbart

et al. 2018

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Summary The continuous growth of supercomputers is accompanied by increased complexity of the intra‐node level and the interconnection topology. Consequently, the whole software stack ranging from the system software to the applications has to evolve, eg, by means of fault tolerance and support for the rising intra‐node parallelism. Migration techniques are one means to address these challenges. On the one hand, they facilitate the maintenance process by enabling the evacuation of individual nodes during runtime, ie, the implementation of fault avoidance. On the other hand, they enable dynamic load balancing for an improvement of the system's efficiency. However, these prospects come along with certain challenges. On the process level, migration mechanisms have to resolve so‐called residual dependencies to the source node, eg, the communication hardware. On the job level, migrations affect the communication topology, which should be addressed by the communication stack, ie, the optimal communication path between a pair of processes might change after a migration. In this article, we explore migration mechanisms for HPC and discuss their prospects as well as the challenges. Furthermore, we present solutions enabling their efficient usage in this domain. Finally, we evaluate our prototype co‐scheduler leveraging migration for workload optimization.

show abstract

Section: A Protocol Enabling the Migration Of Mpi Processesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Prospects and challenges of virtual machine migration in HPC

Pickartz

Clauss

Breitbart

et al. 2018

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…The authors propose a monitoring-based weakly consistent resource stealing technique to achieve resource balancing in distributed high performance computing. In [14], authors investigate migration of applications for high performance computing by deriving respective requirements for specific field of applications. In [10], authors consider practical parameters such as communication delay and propose an optimized dynamic load balancing algorithm for exasclae distributed computing systems.…”

Section: Related Researchesmentioning

confidence: 99%

“…In the next-generation high performance computing (HPC) systems, the scientific programs are going to get more complex with unpredictable requests what requires large scale and more powerful computing systems [3]. Distributed computing system (DCS) is a promising solution to such demands.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Load Balancing Dashboard in Dynamic Distributed Systems

Mirtaheri

Fatemi

Grandinetti

2017

JSFI

View full text Add to dashboard Cite

Considering the dynamic nature of new generation scientific problems, load balancing is a necessity to manage the load in an efficient manner. Load balancing systems are used to optimize the resource consumption, maximize the throughput, minimize response time, and to prevent overload in resources. In current research, we consider operational distributed systems with dynamic variables caused by different nature of the applications and heterogeneity of the various levels in the system. The conducted studies indicate that many different factors should be considered to select the load balancing algorithm, including the processing power, load transfer and communication delay of nodes. In this work, We aim to design a dashboard that is capable of merging the load balancing algorithms in different environments. We design an adaptive system infrastructure with the ability to adjust various factors in the run time of a load balancing algorithm. We propose a task and a resource allocation mechanism and further introduce a mathematical model of load balancing process in the system. We calculate a normalized hardware score that determines the maturity of the system according to the environmental conditions of the load balancing process. The evaluation results confirm that the proposed method performs well and reduces the probability of system failure.

show abstract