Implementation and performance analysis of non-blocking collective operations for MPI

Abstract

Collective operations and non-blocking point-to-point operations have always been part of MPI. Although non-blocking collective operations are an obvious extension to MPI, there have been no comprehensive studies of this functionality. In this paper we present LibNBC, a portable high-performance library for implementing non-blocking collective MPI communication operations. LibNBC provides non-blocking versions of all MPI collective operations, is layered on top of MPI-1, and is portable to nearly all parallel architectures. To measure the performance characteristics of our implementation, we also present a microbenchmark for measuring both latency and overlap of computation and communication. Experimental results demonstrate that the blocking performance of the collective operations in our library is comparable to that of collective operations in other highperformance MPI implementations. Our library introduces a very low overhead between the application and the underlying MPI and thus, in conjunction with the potential to overlap communication with computation, offers the potential for optimizing real-world applications.

Towards Efficient MapReduce Using MPI

“…However, to retain the benefits of optimized collective operations, we would need a nonblocking version of those operations to enable overlap. Such nonblocking collective operations are proposed for MPI-3 [15] and a reference implementation exists with LibNBC [16]. We can easily apply nonblocking collective operations to the map as well as the reduce functionality.…”

Communication Optimization for Medical Image Reconstruction Algorithms

“…In order to optimize the parallel algorithm, we reduce the overhead arising from the allreduce step by overlapping its communication with computations that are independent of the communicated data. We use LibNBC's [6] non-blocking version of MPI Allreduce called NBC Iallreduce, and the MPI Wait counterpart NBC Wait.…”

“…LibNBC's allreduce uses multiple communication rounds (cf. [6]). This requires the user to ensure progress manually by calling NBC Test or run a separate thread that manages the progression of LibNBC (i.e., progress thread).…”

“…In our case study, the scalability for the fixed problem size (strong scaling) is limited by a collective data reduction operation in which the message size is independent of the number of MPI processes (in our example 48 M iB). To reduce the communication overhead, we transform our code to leverage non-blocking collective operations offered by LibNBC [6], which provide-additionally to the overlapping of communication with computation-high-level communication offload using the InfiniBand network. We analyze the code transformations and provide an analytical runtime model that identifies the overlap potential of our approach.…”

LogGP in theory and practice – An in-depth analysis of modern interconnection networks and benchmarking methods for collective operations

“…This distinction between CPU overhead and network parameters enables researchers to model overlap of communication and computation efficiently. We use this ability to assess the overlap potential of different network interconnect architectures and to optimize the implementation of our non-blocking collective operations library LibNBC [12]. The models of the LogP family have been used by different research groups to derive new algorithms for parallel computing, predict the performance of existing algorithms, or prove an algorithm's optimality [13][14][15][16][17][18].…”