MPI Detach - Asynchronous Local Completion

Abstract

When aiming for large scale parallel computing, waiting time due to network latency, synchronization, and load imbalance are the primary opponents of high parallel efficiency. A common approach to hide latency with computation is the use of non-blocking communication. In the presence of a consistent load imbalance, synchronization cost is just the visible symptom of the load imbalance. Tasking approaches as in OpenMP, TBB, OmpSs, or C++20 coroutines promise to expose a higher degree of concurrency, which can be distributed on available execution units and significantly increase load balance. Available MPI non-blocking functionality does not integrate seamlessly into such tasking parallelization. In this work, we present a slim extension of the MPI interface to allow seamless integration of non-blocking communication with available concepts of asynchronous execution in OpenMP and C++.

Communication-Aware Task Scheduling Strategy in Hybrid MPI+OpenMP Applications

“…MPI+OpenMP(tasks) model may lead to a loss of thread, when a thread executes blocking MPI code within an OpenMP task [11,12]. Many works addressed this issue [4,9,11,16,18,19]. Some approaches [4,11] consist of marking communication tasks from user codes and dedicating threads to communication or computation.…”

“…Schuchart et al [19] explored various implementations of it, and having implementations that effectively suspend, enables the expression of fine MPI data movement within OpenMP tasks. This resulted in a more efficient implementation of the blocked Cholesky factorization with fewer synchronizations and led to new approaches on MPI+OpenMP(tasks) interoperability, such as TAMPI [18] and MPI_Detach [16]. TAMPI was proposed as a user library to enable blocking-tasks pause and resume mechanism.…”

“…It transforms calls to MPI blocking operations to non-blocking ones through the PMPI interface and interoperates with the underlying tasking runtime -typically using the taskyield in OpenMP, or nanos6_block_current_task in Nanos6. The authors of [16] proposed another interoperability approach using the detach clause, which implies MPI specifications extensions to add asynchronous callbacks on communications completion, and also user code adaptations. Among all these works, our solution on the loss of threads issue differs from [4,11,16]: we aim at no user code modifications, and to progress both communications and computations by any thread opportunistically.…”

“…The authors of [16] proposed another interoperability approach using the detach clause, which implies MPI specifications extensions to add asynchronous callbacks on communications completion, and also user code adaptations. Among all these works, our solution on the loss of threads issue differs from [4,11,16]: we aim at no user code modifications, and to progress both communications and computations by any thread opportunistically. Our approach is more likely a mix of [9,16,18] with automation through runtime interoperations.…”

“…Deadlocks can be due to the loss of cores when threads execute blocking MPI calls within OpenMP tasks [11]. Several solutions address this issue [16,18,20] and enable working MPI+OpenMP(tasks) codes, but performance issues remain. Task scheduling in this hybrid context can significantly improve the overall performance.…”