Formal Modeling and Performance Evaluation of a Run-Time Rank Remapping Technique in Broadcast, Allgather and Allreduce MPI Collective Operations

Task Packing: Efficient task scheduling in unbalanced parallel programs to maximize CPU utilization

“…In addition, this collective has a hierarchical design adapted to the target platform. The reduction is solved following a specific rank ordering in this hierarchical design [63] generating dependencies between intermediate calculations. This fact combined with several tasks sharing one CPU (oversubscription) may degrade performance proportional to the size of the application.…”

Sparbit: a new logarithmic-cost and data locality-aware MPI Allgather algorithm

“…On every step, a process with rank r will send a block to the process with rank r+1 and receive another from the process with rank r−1 (wrapping around if a destination or source is out of bounds). Each process's own block is sent on the first step, while on all others the block received on the previous step is forwarded [7]. Hereafter, the number of processes involved in the algorithm is represented by p, while m represents the total amount of data that a process must have at the end of the operation.…”

“…The formal definitions of the algorithms assume equally balanced communication costs to all peers, but computing clusters and supercomputers often employ hierarchical network topologies [6]. On these networks the cost for performing communication between two nodes is highly dependent on the physical location of each peer [7], and the further away they are, the longer are the physical paths between them and therefore the higher the latency. From a bandwidth perspective, the further away two nodes are the higher is the chance that their communication will cross the core of the network, whose bandwidth is more expensive and supports less saturation than the edge [17], possibly leading to slowdowns or contentions.…”

“…The delay magnitude for the completion of a collective is a product of several factors, which include the underlying hardware topology [5], communication protocols, network capacity [3], placement of processes [6], [7] and others. However, one of paramount importance is the performance of the algorithm employed to coordinate the high level inter process communication and block transferences [8], [9].…”

“…For Allgather, the generally available algorithms are Ring, Neighbor Exchange, Bruck and Recursive Doubling. The Ring algorithm has a linear growth of both latency and bandwidth time with the increase in the number of processes [7]. Neighbor Exchange has the same asymptotic behaviour but with a less steep increase in time, requiring only half of the Ring's steps, with the downside of only working for even numbers of processes [11].…”