Enabling callback-driven runtime introspection via MPI_T

Abstract

Understanding the behavior of parallel applications that use the Message Passing Interface (MPI) is critical for optimizing communication performance. Performance tools for MPI currently rely on the PMPI Profiling Interface or the MPI Tools Information Interface, MPI_T, for portably collecting information for performance measurement and analysis. While tools using these interfaces have proven to be extremely valuable for performance tuning, these interfaces only provide synchronous information, i.e., when an MPI or an MPI_T function is called. There is currently no option for collecting information about asynchronous events from within the MPI library. In this work we propose a callback-driven interface for event notification from MPI implementations. Our approach is integrated in the existing MPI_T interface and provides a portable API for tools to discover and register for events of interest. We demonstrate the functionality and usability of the interface with a prototype implementation in Open MPI, a small logging tool (MEL) and the measurement infrastructure Score-P. CCS CONCEPTS• General and reference → Measurement; Performance; • Software and its engineering → Software libraries and repositories; KEYWORDS MPI, callback functions, runtime introspection ACM Reference Format:

Optimizing computation-communication overlap in asynchronous task-based programs

“…Our approach proposes a set of events triggered by MPI and captured by ATaP runtime systems. To be consistent with the MPI standard, we implement our techniques on top of existing solutions like MPI T, the MPI Tool Information interface introduced in MPI 3.0 [11], as well as the recently proposed MPI T Events extensions [13]. The latter provides the necessary infrastructure for callbacks in MPI, intended for the support of tracing tools, but does not define any concrete events matching the philosophy of MPI T. In particular, we propose adding the following events to MPI: • MPI COLLECTIVE PARTIAL INCOMING signals the arrival of some data in the context of a collective communication.…”

“…In particular, by having the events described in Section 3.1 handled by callbacks, we release the ATaP runtime system from the need for polling the event queue. For this functionality, we directly rely on the MPI T Events proposal [13], which provides generic callbacks mainly intended to implement tracing tools. We use it to track the events described in Section 3.1 and notify the ATaP runtime, which can then associate a handler function by invok- ing the MPI T Event handle alloc call, as described by Hermanns et al [13].…”

“…For this functionality, we directly rely on the MPI T Events proposal [13], which provides generic callbacks mainly intended to implement tracing tools. We use it to track the events described in Section 3.1 and notify the ATaP runtime, which can then associate a handler function by invok- ing the MPI T Event handle alloc call, as described by Hermanns et al [13]. Once invoked, the runtime receives an MPI events object, which can be decoded with MPI T Event read.…”

“…We present two mechanisms, similar to the MPI tools interface (MPI T) [13], for exchanging information between MPI and an ATaP runtime system and analyze their trade-offs: 1. a fast mechanism to poll events when idle using a lock-free queue, and 2. a delivery solution based on callbacks that can benefit from a hardware implementation, shown in the bottom row of Figure 1. These mechanisms allow ATaP runtimes to seamlessly interoperate with MPI by reducing or completely eliminating the need for explicit polling or waiting on specific requests, and instead deliberately invoking the progress engine only when needed, driven by runtime events.…”

Optimizing computation-communication overlap in asynchronous task-based programs

“…Our approach proposes a set of events triggered by MPI and captured by ATaP runtime systems. To be consistent with the MPI standard, we implement our techniques on top of existing solutions like MPI T, the MPI Tool Information interface introduced in MPI 3.0 [11], as well as the recently proposed MPI T Events extensions [13]. The latter provides the necessary infrastructure for callbacks in MPI, intended for the support of tracing tools, but does not define any concrete events matching the philosophy of MPI T. In particular, we propose adding the following events to MPI:…”