A study of process arrival patterns for MPI collective operations

Abstract

Process arrival pattern, which denotes the timing when different processes arrive at an MPI collective operation, can have a significant impact on the performance of the operation. In this work, we characterize the process arrival patterns in a set of MPI programs on two common cluster platforms, use a micro-benchmark to study the process arrival patterns in MPI programs with balanced loads, and investigate the impacts of different process arrival patterns on collective algorithms. Our results show that (1) the differences between the times when different processes arrive at a collective operation are usually sufficiently large to affect the performance; (2) application developers in general cannot effectively control the process arrival patterns in their MPI programs in the cluster environment: balancing loads at the application level does not balance the process arrival patterns; and (3) the performance of collective communication algorithms is sensitive to process arrival patterns. These results indicate that process arrival pattern is an important factor that must be taken into consideration in developing and optimizing MPI collective routines. We propose a scheme that achieves high performance with different process arrival patterns, and demonstrate that by explicitly considering process arrival pattern, more efficient MPI collective routines than the current ones can be obtained.

Process Arrival Pattern Aware Alltoall and Allgather on InfiniBand Clusters

“…For the larger class C, the imbalance factors are even larger. The gathered results on our platform confirm previous studies in [2,3] that processes arrive at collective sites at different times.…”

“…However, recent studies [2,3] have shown that processes in real applications can arrive at the collective calls at different times. This imbalanced process arrival pattern can significantly affect the performance of the collective operations [2]. In addition, it has been found that different collective communication algorithms react differently to PAP [2].…”

“…This imbalanced process arrival pattern can significantly affect the performance of the collective operations [2]. In addition, it has been found that different collective communication algorithms react differently to PAP [2]. In this regard, the authors in [3] have recently proposed PAP aware broadcast algorithms and implemented them using MPI point-to-point primitives.…”

“…Most research on developing and implementing efficient collective communication algorithms assume all MPI processes involved in the operation arrive at the same time at the collective call. However, recent research has shown that Process Arrival Pattern (PAP) for collective operations are adequately imbalanced that will adversely affect the performance of collective communications [2]. In addition, different collective communication algorithms react differently to process arrival patterns [2].…”

“…However, recent research has shown that Process Arrival Pattern (PAP) for collective operations are adequately imbalanced that will adversely affect the performance of collective communications [2]. In addition, different collective communication algorithms react differently to process arrival patterns [2]. In this regard, the authors in [3] have recently proposed process arrival pattern aware MPI-Bcast() algorithms and implemented them using MPI point-to-point primitives.…”