Hot-Spot Avoidance With Multi-Pathing Over InfiniBand: An MPI Perspective

Abstract

Large scale InfiniBand clusters are becoming increasingly popular, as reflected by the TOP 500 Supercomputer rankings. At the same time, fat tree has become a popular interconnection topology for these clusters, since it allows multiple paths to be available in between a pair of nodes. However, even with fat tree, hot-spots may occur in the network depending upon the route configuration between end nodes and communication pattern(s) in the application. To make matters worse, the deterministic routing nature of In-finiBand limits the application from effective use of multiple paths transparently and avoid the hot-spots in the network. Simulation based studies for switches and adapters to implement congestion control have been proposed in the literature. However, these studies have focussed on providing congestion control for the communication path, and not on utilizing multiple paths in the network for hot-spot avoidance. In this paper, we design an MPI functionality, which provides hot-spot avoidance for different communications, without a priori knowledge of the pattern. We leverage LMC (LID Mask Count) mechanism of InfiniBand to create multiple paths in the network and present the design issues (scheduling policies, selecting number of paths, scalability aspects) of our design. We implement our design and evaluate it with Pallas collective communication and MPI applications. On an InfiniBand cluster with 48 processes, collective operations like MPI All-to-all Personalized and MPI Reduce Scatter show an improvement of 27% and 19% respectively. Our evaluation with MPI applications like NAS Parallel Benchmarks and PSTSWM on 64 processes shows significant improvement in execution time with this functionality.

Adaptive Routing Strategies for Modern High Performance Networks

“…Thus, it is comparable to the solution discussed by Vishnu et al in [19]. However, Vishnu et al only discuss a particular worst-case pattern.…”

“…Martinez et al proposed a fully adaptive routing scheme for InfiniBand networks [10] that requires significant changes to the crossbar switches and thus is currently not implemented. A simpler scheme for InfiniBand which uses multiple endpoint addresses (LIDs) per node has been proposed in [9] and implemented in [19]. Another scheme to optimize routing in fat-tree networks that requires global routing information and a specialized hardware has been proposed in [5].…”

“…One practical implementation that leverages a decentralized routing scheme and does not require global routing knowledge has been implemented for InfiniBand [19]. However, several technical problems, for example the requirement of large routing tables in all switches and the huge number of endpoint contexts (InfiniBand Queue Pairs) render this approach rather impractical for medium to largescale networks.…”

Extending Summation Precision for Network Reduction Operations

“…One way is for nodes to send their partial sum (a single variable) to a single node to perform the summation and generate a single result. This only occupies a single node's processor but completion time grows linearly with the number of operands and can also create network hotspots [17]. The other approach (used in most MPI implementations) performs the summation in a tree fashion, where each node adds a number of operands and passes the partial sum (also a single variable) to the next level of the tree.…”

Analyzing the impact of supporting out-of-order communication on in-order performance with iWARP

“…As high-end computing (HEC) systems continue to increase rapidly in size, their network subsystems must scale as well, particularly to address issues such as hot-spot congestion [22], [26] and hardware faults [8]. Multi-path communication, supported by InfiniBand (IB) [1] and 10-Gigabit Ethernet (10GE) [16], provides a way to address these issues.…”