PAMI: A Parallel Active Message Interface for the Blue Gene/Q Supercomputer

Abstract

The Blue Gene/Q machine is the next generation in the line of IBM massively parallel supercomputers, designed to scale to 262144 nodes and sixteen million threads. With each BG/Q node having 68 hardware threads, hybrid programming paradigms, which use message passing among nodes and multi-threading within nodes, are ideal and will enable applications to achieve high throughput on BG/Q. With such unprecedented massive parallelism and scale, this paper is a groundbreaking effort to explore the design challenges for designing a communication library that can match and exploit such massive parallelism In particular, we present the Parallel Active Messaging Interface (PAMI) library as our BG/Q library solution to the many challenges that come with a machine at such scale. PAMI provides (1) novel techniques to partition the application communication overhead into many contexts that can be accelerated by communication threads; (2) client and context objects to support multiple and different programming paradigms; (3) lockless algorithms to speed up MPI message rate; and (4) novel techniques leveraging the new BG/Q architectural features such as the scalable atomic primitives implemented in the L2 cache, the highly parallel hardware messaging unit that supports both point-to-point and collective operations, and the collective hardware acceleration for operations such as broadcast, reduce, and allreduce. We experimented with PAMI on 2048 BG/Q nodes and the results show high messaging rates as well as low latencies and high throughputs for collective communication operations.

Parallel Combinatorial Optimization with Decision Diagrams

“…The machines are connected via a network that supports the PAMI message passing interface [21], although DDX10 can also be easily compiled to run using the usual network communication with TCP sockets. We used 24 workers on each machine, using as many machines as necessary to operate w workers in parallel.…”

Enabling MPI interoperability through flexible communication endpoints

“…An advantage of the static endpoints scheme is that it can better align MPI implementations with systems where endpoint resources are reserved or created when MPI is initialized, for example, in the case of implementations based on IBM's PAMI low-level communication library [11]. However, statically specifying the number of endpoints restricts the ability of MPI to interoperate fully with dynamic threading models and limits opportunities for libraries to use endpoints.…”

“…Such networks may be unable to create create additional endpoints later, such as an implementation of MPI over PAMI [11]. In this case, the implementation may create all endpoints when MPI_Init is called, possibly directed by runtime parameters or environment variables.…”

“…For example, endpoint ranks may be implemented through distinct network resources, such as NICs, DMA FIFOs, or PAMI Contexts [11]. An MPI implementation might have an optimal number of endpoints based on the number of available hardware resources; users could query this optimal number and use it to influence the number of endpoints that are created.…”

Scalable Algorithms for Constructing Balanced Spanning Trees on System-Ranked Process Groups

“…This also results in lesser dependencies on remote information for the progress of the collective, which should lead to lesser wait times and faster completion of the operation. We believe this benefit will become more prominent for implementations that exploit one-sided data transfer calls provided by some network messaging APIs [4,9]. Second, our schemes can construct and use communication trees of arbitrary branching factors.…”