Two-level Dynamic Load Balancing for High Performance Scientific Applications

Abstract

Scientific applications are often complex, irregular, and computationally-intensive. To accommodate the ever-increasing computational demands of scientific applications, high performance computing (HPC) systems have become larger and more complex, offering parallelism at multiple levels (e.g., nodes, cores per node, threads per core). Scientific applications need to exploit all the available multilevel hardware parallelism to harness the available computational power. The performance of applications executing on such HPC systems may adversely be affected by load imbalance at multiple levels, caused by problem, algorithmic, and systemic characteristics. Nevertheless, most existing load balancing methods do not simultaneously address load imbalance at multiple levels. This work investigates the impact of load imbalance on the performance of three scientific applications at the thread and process levels. We jointly apply and evaluate selected dynamic loop self-scheduling (DLS) techniques to both levels. Specifically, we employ the extended LaPeSD OpenMP runtime library [1] at the thread level, and extend the DLS4LB MPI-based dynamic load balancing library [2] at the process level. This approach is generic and applicable to any multiprocess-multithreaded computationally-intensive application (programmed using MPI and OpenMP). We conduct an exhaustive set of experiments to assess and compare six DLS techniques at the thread level and eleven at the process level. The results show that improved application performance, by up to 21%, can only be achieved by jointly addressing load imbalance at the two levels. We offer insights into the performance of the selected DLS techniques and discuss the interplay of load balancing at the thread level and process level.

LB4OMP: A Dynamic Load Balancing Library for Multithreaded Applications

“…It has been recently shown that thread-level load imbalance has a significant impact on the performance of hybrid MPI+OpenMP applications [13]. OpenMP is the most widely-used standard for expressing and exploiting nodelevel parallelism.…”

“…This work builds on recent work on multilevel load balancing [13] by concentrating on thread-level scheduling and deepening the analysis of its performance impact for multithreaded applications executing on hierarchical parallel systems. Specifically, this work provides a broad range of dynamic loop self-scheduling (DLS) techniques, implemented in a unified OpenMP runtime library (RTL), called LB4OMP [17] that can readily be used for MPI+OpenMP applications.…”

A Distributed Chunk Calculation Approach for Self-scheduling of Parallel Applications on Distributed-memory Systems

“…LB4MPI extends the LB tool [13] by including certain bug fixes and additional DLS techniques. LB4MPI has been used to enhance the performance of various scientific applications [31]. In this work, we extend the LB4MPI in two directions: (1) We enable the support of DCA.…”

LB4OMP: A Dynamic Load Balancing Library for Multithreaded Applications

“…It has been recently shown that thread-level load imbalance has a significant impact on the performance of hybrid MPI+OpenMP applications [11]. OpenMP is the most widely-used standard for expressing and exploiting node-level parallelism.…”

“…This work builds on recent work on multilevel load balancing [11] by concentrating on thread-level scheduling and deepening the analysis of its performance impact for multithreaded applications executing on hierarchical parallel systems. Specifically, this work provides a broad range of dynamic loop self-scheduling (DLS) techniques, implemented in a unified OpenMP runtime library (RTL), called LB4OMP 3 that can readily be used for MPI+OpenMP applications.…”