Load Balancing Scientific Applications

Abstract

The largest supercomputers have millions of independent processors, and concurrency levels are rapidly increasing. For ideal efficiency, developers of the simulations that run on these machines must ensure that computational work is evenly balanced among processors. Assigning work evenly is challenging because many large modern parallel codes simulate behavior of physical systems that evolve over time, and their workloads change over time. Furthermore, the cost of imbalanced load increases with scale because most large-scale scientific simulations today use a Single Program Multiple Data (SPMD) parallel programming model, and an increasing number of processors will wait for the slowest one at the synchronization points. To address load imbalance, many large-scale parallel applications use dynamic load balance algorithms to redistribute work evenly. The research objective of this dissertation is to develop methods to decide when and how to load balance the application, and to balance it effectively and affordably. We measure and evaluate the computational load of the application, and develop strategies to decide when and how to correct the imbalance. Depending on the simulation, a fast, local load balance algorithm may be suitable, or a more sophisticated and expensive algorithm may be required. We developed a model for comparison of load balance algorithms for a specific state of the simulation that enables the selection of a balancing algorithm that will minimize overall runtime. Dynamic load balancing of parallel applications becomes more critical at scale, while also being expensive. To make the load balance correction affordable at scale, we propose a lazy load balancing strategy that evaluates the imbalance and computes the new assignment of work to processes asynchronously to the main application computation. We decouple the load balance algorithm from the application and run it on potentially fewer, separate I would like to thank the students and faculty of Western Oregon University.

On the Benefits of Anticipating Load Imbalance for Performance Optimization of Parallel Applications

“…The LB problem is twofold: (i) find a partitioning of the current workload (W tot (i)) into P partitions, where P is the number of PEs, such that each partition has almost the same workload and (ii) find a mapping of the partitions on the PEs such that the wall clock time to compute those partitions in parallel is minimal [8]. Unfortunately, it has been shown in previous works that the LB problem is fundamentally a balanced graph partitioning problem, which is NP-Complete [9].…”

Towards Informed Partitioning for Load Balancing: a Proof-of-Concept

“…(i.e., at which iteration should I trigger the load balancing mechanism?) [2] to obtain the maximal performance.…”