Topology agnostic hot‐spot avoidance with InfiniBand

Abstract

SUMMARY InfiniBand has become a very popular interconnect due to its advanced features and open standard.Large-scale InfiniBand clusters are becoming very popular, as reflected by the TOP 500 supercomputer rankings. However, even with popular topologies such as constant bi-section bandwidth Fat Tree, hot-spots may occur with InfiniBand due to inappropriate configuration of network paths, presence of other jobs in the network and un-availability of adaptive routing. In this paper, we present a hot-spot avoidance layer (HSAL) for InfiniBand, which provides hot-spot avoidance using path bandwidth estimation and multi-pathing using LMC mechanism, without taking the network topology into account. We propose an adaptive striping policy with batch-based striping and sorting approach, for efficient utilization of disjoint network paths. Integration of HSAL with MPI, the de facto programming model of clusters, shows promising results with collective communication primitives and MPI applications.

dFtree

“…A recent proposal by Rodriguez et al [17] addresses the congestion issue from a routing perspective, but in an application-specific manner and without using virtual lanes (VLs). Another approach using a combination of multipath routing and bandwidth estimation was proposed by Vishnu et al in [19], but this is significantly more complex to implement than our proposal. A third proposal by Escudero-Sahuquillo et al [6] uses multiple queues at the input ports in the switches to avoid HOL blocking, but this is not compatible with any existing network technology and requires new hardware to be built.…”

Fault Modeling of Extreme Scale Applications Using Machine Learning

“…We address these questions in the upcoming sections of the paper. We begin with a description of the three applications, which we have considered for evaluation in this paper -NWChem [9], [12], LULESH [10] and Support Vector Machines (SVM) [11], [13], [3].…”

Final Report for Project DE-FC02-06ER25755 [Pmodels2]

“…To make matters worse, the deterministic routing nature of InfiniBand limits the application from effective use of multiple paths transparently and avoid the hot-spots in the network. To alleviate this situation, we have designed an MPI functionality which provides network agnostic routing, without a-priori knowledge of the pattern [35].…”