RDMA read based rendezvous protocol for MPI over InfiniBand

Abstract

Message Passing Interface (MPI) is a popular parallel programming model for scientific applications. Most high-performance MPI implementations use Rendezvous Protocol for efficient transfer of large messages. This protocol can be designed using either RDMA Write or RDMA Read. Usually, this protocol is implemented using RDMA Write. The RDMA Write based protocol requires a twoway handshake between the sending and receiving processes. On the other hand, to achieve low latency, MPI implementations often provide a polling based progress engine. The two-way handshake requires the polling progress engine to discover multiple control messages. This in turn places a restriction on MPI applications that they should call into the MPI library to make progress. For compute or I/O intensive applications, it is not possible to do so. Thus, most communication progress is made only after the computation or I/O is over. This hampers the computation to communication overlap severely, which can have a detrimental impact on the overall application performance. In this paper, we propose several mechanisms to exploit RDMA Read and selective interrupt based asynchronous progress to provide better computation/communication overlap on InfiniBand clusters. Our evaluations reveal that it is possible to achieve nearly complete computation/communication overlap using our RDMA Read with Interrupt based Protocol. Additionally, our schemes yield around 50% better communication progress rate when computation is overlapped with communication. Further, our application evaluation with Linpack (HPL) and NAS-SP (Class C) reveals that MPI Wait time is reduced by around 30% and 28%, respectively, for a 32 node InfiniBand cluster. We observe that the gains obtained in the MPI Wait time increase as the system size increases. This indicates that our designs have a strong positive impact on scalability of parallel applications.

Intra-Epoch Message Scheduling To Exploit Unused or Residual Overlapping Potential

“…t * / 7 i n t r a n r m a q / * Queue o f d e f e r r e d i n t r a − node RMA * / 8 i n t e r n r m a q / * Queue o f d e f e r r e d i n t e r − node RMA * / 9 p e n d i n g i n t e r n o d e r m a q / * Queue o f i n t e r − node RMA t h a t a r e p o s t e d and pending c o m p l e t i o n * / 10 } 11 / * END DATA AND GLOBAL DEFINITIONS * / 12 13 …”

“…The literature is particularly abundant for communication/computation overlapping for two-sided communications [8,12,13,19]. In order to mitigate the synchronization effects, previous MPI onesided communication designs deferred the blocking steps to the epoch-closing routine [1,14].…”

Profile Guided MPI Protocol Selection for Point-to-Point Communication Calls

“…Yuan et al refine the traditional rendezvous protocol over RDMA [9] presented by others [10], [7], [8] through two protocols they denote receiver initiated and sender initiated. In the case of receiver initiated rendezvous protocol, the receiver notifies the sender of the user buffer's address.…”

A scalable and generic task scheduling system for communication libraries

“…For instance, RDMARead methods allow to cut one message in the rendezvous handshake and permit to seamlessly overlap communication and application computation [10] on the sender side. However this technique only works on a given set of hardware that implements RDMA and only allows to overlap at the sender side.…”