High Performance MPI on IBM 12x InfiniBand Architecture

Vishnu, Abhinav; Benton, Brad; Panda, Dhabaleswar K.

doi:10.1109/ipdps.2007.370407

Cited by 10 publications

(4 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The article presents an indepth performance analysis of leveraging offloaded collective communication capabilities provided by Quadrics Interconnect. Leveraging the communication and scalability primitives provided by InfiniBand, Panda et al have presented design of Message Passing Interface (MPI) using InfiniBand DDR Interconnect, InfiniBand QDR Interconnect and Offloaded collective communication primitives using InfiniBand QDR Interconnect [18], [19], [20].…”

Section: Related Workmentioning

confidence: 99%

Evaluating the Potential of Cray Gemini Interconnect for PGAS Communication Runtime Systems

Vishnu

Bruggencate²,

Olson³

2011

2011 IEEE 19th Annual Symposium on High Performance Interconnects

Self Cite

View full text Add to dashboard Cite

The Cray Gemini Interconnect has been recently introduced as the next generation network for building scalable multi-petascale supercomputers. The Cray XE6 systems, which use the Gemini Interconnect are becoming available with Message Passing Interface (MPI) and Partitioned Global Address Space (PGAS) Models such as as Global Arrays, Unified Parallel C, Co-Array Fortran and Cascade High Performance Language. These PGAS models use one-sided communication runtime systems such as MPI-Remote Memory Access, Aggregate Remote Memory Copy Interface and proprietary communication runtime systems. The primary objective of our work is to study the potential of Cray Gemini Interconnect by designing application specific micro-benchmarks using the DMAPP userspace library. We design micro-benchmarks to study the performance of simple communication primitives and application specific microbenchmarks to understand the behavior of Gemini Interconnect at scale. In our experiments, the Gemini Interconnect can achieve a peak bandwidth of 6911 MB/s and a latency of 1µs for get communication primitive. Scalability tests for atomic memory operations and shift communication operation up to 65536 processes show the efficacy of the Gemini Interconnect.

show abstract

Section: Related Workmentioning

confidence: 99%

Evaluating the Potential of Cray Gemini Interconnect for PGAS Communication Runtime Systems

Vishnu

Bruggencate²,

Olson³

2011

2011 IEEE 19th Annual Symposium on High Performance Interconnects

Self Cite

View full text Add to dashboard Cite

show abstract

“…Many researchers have focused on providing MPI support for multi-rail networks [8,3,12]. In our previous work, we have designed MPI-2 one sided communication using multi-rail InfiniBand networks [14].…”

Section: Related Workmentioning

confidence: 99%

Hot-Spot Avoidance With Multi-Pathing Over InfiniBand: An MPI Perspective

Vishnu

Koop

Moody

et al. 2007

Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid '07)

View full text Add to dashboard Cite

Large scale InfiniBand clusters are becoming increasingly popular, as reflected by the TOP 500 Supercomputer rankings. At the same time, fat tree has become a popular interconnection topology for these clusters, since it allows multiple paths to be available in between a pair of nodes. However, even with fat tree, hot-spots may occur in the network depending upon the route configuration between end nodes and communication pattern(s) in the application. To make matters worse, the deterministic routing nature of In-finiBand limits the application from effective use of multiple paths transparently and avoid the hot-spots in the network. Simulation based studies for switches and adapters to implement congestion control have been proposed in the literature. However, these studies have focussed on providing congestion control for the communication path, and not on utilizing multiple paths in the network for hot-spot avoidance. In this paper, we design an MPI functionality, which provides hot-spot avoidance for different communications, without a priori knowledge of the pattern. We leverage LMC (LID Mask Count) mechanism of InfiniBand to create multiple paths in the network and present the design issues (scheduling policies, selecting number of paths, scalability aspects) of our design. We implement our design and evaluate it with Pallas collective communication and MPI applications. On an InfiniBand cluster with 48 processes, collective operations like MPI All-to-all Personalized and MPI Reduce Scatter show an improvement of 27% and 19% respectively. Our evaluation with MPI applications like NAS Parallel Benchmarks and PSTSWM on 64 processes shows significant improvement in execution time with this functionality.

show abstract

“…Liu et al have studied the designs for providing support for multi-rail InfiniBand clusters with twosided communication [6]. proposed by Vishnu et al [14,20]. IBM HPS architecture supports multiple adapters with connectivity to different switches.…”

Section: Related Studymentioning

confidence: 99%

“…A popular mechanism to alleviate hot-spots in the network is to leverage multi-rail clusters [6,14,18,19]. proposed by Vishnu et al [14,20]. IBM HPS architecture supports multiple adapters with connectivity to different switches.…”

Section: Related Studymentioning

confidence: 99%

Topology agnostic hot‐spot avoidance with InfiniBand

Vishnu¹,

Koop²,

Moody

et al. 2008

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SUMMARY InfiniBand has become a very popular interconnect due to its advanced features and open standard.Large-scale InfiniBand clusters are becoming very popular, as reflected by the TOP 500 supercomputer rankings. However, even with popular topologies such as constant bi-section bandwidth Fat Tree, hot-spots may occur with InfiniBand due to inappropriate configuration of network paths, presence of other jobs in the network and un-availability of adaptive routing. In this paper, we present a hot-spot avoidance layer (HSAL) for InfiniBand, which provides hot-spot avoidance using path bandwidth estimation and multi-pathing using LMC mechanism, without taking the network topology into account. We propose an adaptive striping policy with batch-based striping and sorting approach, for efficient utilization of disjoint network paths. Integration of HSAL with MPI, the de facto programming model of clusters, shows promising results with collective communication primitives and MPI applications.

show abstract

High Performance MPI on IBM 12x InfiniBand Architecture

Cited by 10 publications

References 5 publications

Evaluating the Potential of Cray Gemini Interconnect for PGAS Communication Runtime Systems

Evaluating the Potential of Cray Gemini Interconnect for PGAS Communication Runtime Systems

Hot-Spot Avoidance With Multi-Pathing Over InfiniBand: An MPI Perspective

Topology agnostic hot‐spot avoidance with InfiniBand

Contact Info

Product

Resources

About