Performance analysis of MPI collective operations

Abstract

Previous studies of application usage show that the performance of collective communications are critical for high-performance computing. Despite active research in the field, both general and feasible solution to the optimization of collective communication problem is still missing.In this paper, we analyze and attempt to improve intracluster collective communication in the context of the widely deployed MPI programming paradigm by extending accepted models of point-to-point communication, such as Hockney, LogP/LogGP, and PLogP, to collective operations. We compare the predictions from models against the experimentally gathered data and using these results, construct optimal decision function for broadcast collective. We quantitatively compare the quality of the model-based decision functions to the experimentally-optimal one. Addi-J. Pješivac-Grbović ( ) · T. Angskun · G. tionally, in this work, we also introduce a new form of an optimized tree-based broadcast algorithm, splitted-binary.Our results show that all of the models can provide useful insights into various aspects of the different algorithms as well as their relative performance. Still, based on our findings, we believe that the complete reliance on models would not yield optimal results. In addition, our experimental results have identified the gap parameter as being the most critical for accurate modeling of both the classical point-topoint-based pipeline and our extensions to fan-out topologies.

LogGP in theory and practice – An in-depth analysis of modern interconnection networks and benchmarking methods for collective operations

“…Both algorithms are used in practice depending on the proportion of P and the message size s (cf. [9]). Figure 7 shows the influence of pipelining on a measurement with P = 16 and n = 5.…”

“…Other models are general-purpose and attempt to be architectureindependent, for example the PRAM [4,5], BSP [6], or the LogP [7] models. Several studies have compared the accuracy of those models [8,9]. In general, it seems that the LogP model family bridges the complexity of real-world interconnection networks and the usability of an abstract network model fairly well.…”

Bandwidth-Latency Models (BSP, LogP)

“…In [8], this has been demonstrated for MPI's collective communication operations, even for hierarchical communi- cation networks with different sets of performance parameters. Pješivac-Grbović et al [10] have shown that PLogP provides flexible and accurate performance modeling.…”

A Case for Non-blocking Collective Operations

“…The CPU idle time during the communication will be modelled and benchmarked. Precise models for collective operations are presented in [20] and for barrier synchronization in [21]. Both studies show that the LogP [22] or LogGP [23] model is able to predict the communication time sufficiently accurately if the processes enter the collective operation simultaneously.…”

“…We assume the dissemination principle to perform MPI BARRIER (1), analyzed in [21]. Our model for MPI ALLREDUCE (2) assumes a simple binomial tree reduce implementation followed by MPI BCAST and our MPI BCAST (3) model assumes a binomial tree implementation (compare proposed models in [20]). …”