Optimizing blocking and nonblocking reduction operations for multicore systems: Hierarchical design and implementation

Abstract

Many scientific simulations, using the Message Passing Interface (MPI) programming model, are sensitive to the performance and scalability of reduction collective operations such as MPI Allreduce and MPI Reduce. These operations are the most widely used abstractions to perform mathematical operations over all processes that are part of the simulation. In this work, we propose a hierarchical design to implement the reduction operations on multicore systems. This design aims to improve the efficiency of reductions by 1) tailoring the algorithms and customizing the implementations for various communication mechanisms in the system 2) providing the ability to configure the depth of hierarchy to match the system architecture, and 3) providing the ability to independently progress each of this hierarchy. Using this design, we implement MPI Allreduce and MPI Reduce operations (and its nonblocking variants MPI Iallreduce and MPI Ireduce) for all message sizes, and evaluate on multiple architectures including InfiniBand and Cray XT5. We leverage and enhance our existing infrastructure, Cheetah, which is a framework for implementing hierarchical collective operations to implement these reductions.The experimental results show that the Cheetah reduction operations outperform the production-grade MPI implementations such as Open MPI default, Cray MPI, and MVAPICH2, demonstrating its efficiency, flexibility and portability. On Infini-Band systems, with a microbenchmark, a 512-process Cheetah nonblocking Allreduce and Reduce achieves a speedup of 23x and 10x, respectively, compared to the default Open MPI reductions. The blocking variants of the reduction operations also show similar performance benefits. A 512-process nonblocking Cheetah Allreduce achieves a speedup of 3x, compared to the default MVAPICH2 Allreduce implementation. On a Cray XT5 system, a 6144-process Cheetah Allreduce outperforms the Cray MPI by 145%. The evaluation with an application kernel, Conjugate Gradient solver, shows that the Cheetah reductions speeds up total time to solution by 195%, demonstrating the potential benefits for scientific simulations.Currently used algorithms and implementations for Allreduce and Reduce suffer from several performance drawbacks on multicore systems. These systems typically consist of tens of Central Processing Unit (CPU) cores on a node, network interface with bandwidth of tens of Giga bytes per second and latency of a few microseconds, and have multiple communication mechanisms -multiple cache levels, intra-node communication buses, and network interfaces -with varying performance characteristics. The multicore system architecture is ubiquitous in extreme scale systems. Also, these systems are widely used by scientific community for executing the scientific simulations [1]. Most existing Allreduce and Reduce implementations do not consider these performance variations in communication mechanisms in modern systems, and typically have a single implementation for all these different communication mechanisms resulting in ...

Energy-Efficient Collective Reduce and Allreduce Operations on Distributed GPUs

“…However, several work has be done in optimizing this trees for MPI, eg. [15] [4], and this is not the scope of this work. However, is worth to mention that these structures can also be used for GGAS and GPU clusters.…”

“…To name a few, in [4], [15] and [3], blocking and nonblocking allreduce and reduce operations are optimized. Also, since GPUs are highly suitable to perform parallel reductions, the in-core reduction on a single GPU has been highly optimized, like described in [16].…”

“…For instance, [2] and [3] show that MPI-based scientific simulations spend over 40% of their run time for reduction operations. Iterative solvers such as Conjugate Gradient, GMRES, and Newton, which are important components of many scientific simulations, highly rely on reductions [4].…”

Towards A Data Centric System Architecture: SHARP

“…Algorithmic work performed by Venkata et al [33] developed short vector blocking and non blocking reduction and barrier operations using a recursive K-ing type host-based approach, and extended work by Thakur [31]. Vadhiar et al [32] presented implementations of blocking reduction, gather and broadcast operations using sequential, chain, binary, binomial tree and Rabenseifner algorithms.…”

Hierarchical redesign of classic MPI reduction algorithms

“…According to research studies over the past two decades [2,3], MPI reduction operations, particularly MPI reduce and allreduce, are the most used collective operations in scientific applications. In the reduce operation, each node i owns a vector x i of n elements.…”