Designing Non-blocking Allreduce with Collective Offload on InfiniBand Clusters: A Case Study with Conjugate Gradient Solvers

Abstract

Abstract-Scientists across a wide range of domains increasingly rely on computer simulation for their investigations. Such simulations often spend a majority of their run-times solving large systems of linear equations that require vast amounts of computational power and memory. It is hence critical to design solvers in a highly efficient and scalable manner. Hypre is a high performance, scalable software library that offers several optimized linear solver routines and pre-conditioners. In this paper, we study the characteristics of Hypre's Preconditioned Conjugate Gradient (PCG) solver algorithm. The PCG routine is known to spend a majority of its communication time in the MPI Allreduce operation to compute a global summation during the innerproduct operation. The MPI Allreduce is a blocking operation whose latency is often a limiting factor to the overall efficiency of the PCG solver routine, and correspondingly the performance of simulations that rely on this solver. Hence, hiding the latency of the MPI Allreduce operation is critical towards scaling the PCG solver routine and improving the performance of many simulations.The upcoming revision of MPI, MPI-3, will provide support for non-blocking collective communication to enable latency-hiding. The latest InfiniBand adapter from Mellanox, ConnectX-2, enables offloading of generalized lists of communication operations to the network interface. Such an interface can be leveraged to design non-blocking collective operations. In this paper, we design fully functional, scalable algorithms for the MPI Iallreduce operation, based on the network offload technology. To the best of our knowledge, this is the first such design to be presented in the literature. Our designs scale beyond 512 processes and we achieve near perfect communication/computation overlap. We also re-design the PCG solver routine to leverage our proposed MPI Iallreduce operation to hide the latency of the global reduction operations. We observe up to 21% improvements in the run-times of the PCG routine, when compared to the default PCG implementation in Hypre. We also note that about 16% of the overall benefits are due to overlapping the Allreduce operations.

Brief introduction of TianHe exascale prototype system

“…The link rate is upgraded to 25 Gbps from 14 Gbps of the TianHe-2A supercomputer system. Collective offload [5] accelerates collective operations, effectively improves the throughput of a single chip. HFI-E provides the software-hardware interface for accessing the high-performance network, implementing the proprietary Mini Packet/Remote Direct Memory Access (MP/RDMA) communication and collective offload mechanism.…”

Optimization of MPI collective operations on the IBM Blue Gene/Q supercomputer

“…In addition, all PAMI collective calls are non-blocking. We plan to explore MPI 3.0 non-blocking (Hoefler et al, 2007;Kandalla et al, 2012Kandalla et al, , 2013 collective implementation that takes advantage of the non-blocking APIs in PAMI.…”

Optimizing blocking and nonblocking reduction operations for multicore systems: Hierarchical design and implementation

“…The mechanisms are generic enough to implement both blocking and nonblocking semantics, rootbased (Reduce) and non-root based reductions (Allreduce), unlike this research [8]. Compared to [5], the concepts and mechanisms are portable across different architectures.…”