Improving Performance Models for Irregular Point-to-Point Communication

Bienz, Amanda; Gropp, William; Olson, Luke N.

doi:10.1145/3236367.3236368

Cited by 11 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ppn represents the number of processors participating in communication. (The Blue Waters data was initially presented in [4] and is replotted here. )…”

Section: Optimal Communication For Blocked Datamentioning

confidence: 99%

Performance Analysis and Optimal Node-Aware Communication for Enlarged Conjugate Gradient Methods

Lockhart¹,

Bienz²,

Gropp³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Krylov methods are a key way of solving large sparse linear systems of equations, but suffer from poor strong scalabilty on distributed memory machines. This is due to high synchronization costs from large numbers of collective communication calls alongside a low computational workload. Enlarged Krylov methods address this issue by decreasing the total iterations to convergence, an artifact of splitting the initial residual and resulting in operations on block vectors. In this paper, we present a performance study of an Enlarged Krylov Method, Enlarged Conjugate Gradients (ECG), noting the impact of block vectors on parallel performance at scale. Most notably, we observe the increased overhead of point-to-point communication as a result of denser messages in the sparse matrix-block vector multiplication kernel. Additionally, we present models to analyze expected performance of ECG, as well as, motivate design decisions. Most importantly, we introduce a new point-to-point communication approach based on node-aware communication techniques that increases efficiency of the method at scale.

show abstract

“…ppn represents the number of processors participating in communication. (The Blue Waters data was initially presented in [4] and is replotted here. )…”

Section: Optimal Communication For Blocked Datamentioning

confidence: 99%

Performance Analysis and Optimal Node-Aware Communication for Enlarged Conjugate Gradient Methods

Lockhart¹,

Bienz²,

Gropp³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Figure 9 displays the cost of sending a single message between two nodes, with various numbers of active processes. Additional parameters not included in the max-rate model add to the cost of communication, such as queue search costs, which result from sending a large number of messages at once, and network contention, which occurs when many processes communicate large amounts of data across multiple links of the network (Bienz et al, 2018). The large costs associated with internode messages motivate replacing them with extra intra-node messages when possible.…”

Section: Node-aware Communicationmentioning

confidence: 99%

Reducing communication in algebraic multigrid with multi-step node aware communication

Bienz

Gropp

Olson

2020

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

Algebraic multigrid (AMG) is often viewed as a scalable [Formula: see text] solver for sparse linear systems. Yet, AMG lacks parallel scalability due to increasingly large costs associated with communication, both in the initial construction of a multigrid hierarchy and in the iterative solve phase. This work introduces a parallel implementation of AMG that reduces the cost of communication, yielding improved parallel scalability. It is common in Message Passing Interface (MPI), particularly in the MPI-everywhere approach, to arrange inter-process communication, so that communication is transported regardless of the location of the send and receive processes. Performance tests show notable differences in the cost of intra- and internode communication, motivating a restructuring of communication. In this case, the communication schedule takes advantage of the less costly intra-node communication, reducing both the number and the size of internode messages. Node-centric communication extends to the range of components in both the setup and solve phase of AMG, yielding an increase in the weak and strong scaling of the entire method.

show abstract

“…However, the cost of communication varies greatly with intranode communication requiring significantly less cost than inter-node. Therefore, the performance model can better capture cost by splitting up intra-and inter-node costs [11], yielding Fig. 8.…”

Section: Node-aware Performance Modelingmentioning

confidence: 99%

“…Node-agnostic performance models, such as the postal model, fail to accurately capture the costs associated with inter-node communication. This model can be improved by splitting communication into intra-and inter-node as well as adding injection bandwidth limits [11], [12].…”

Section: A Node-awarenessmentioning

confidence: 99%

“…Figure 5 displays the modeled cost of sending a single message containing of various sizes on Blue Waters, a Cray supercomputer at the National Center for Supercomputing Applications [2], [3]. The costs were calculated with the max-rate model using parameters measured through ping-pong tests [11], [12]. Intra-socket messages, Fig.…”

Section: Node-aware Parallel Allreduce (Napallreduce)mentioning

confidence: 99%

See 1 more Smart Citation

Node-Aware Improvements to Allreduce

Bienz

Olson

Gropp

2019

2019 IEEE/ACM Workshop on Exascale MPI (ExaMPI)

Self Cite

View full text Add to dashboard Cite

The MPI_Allreduce collective operation is a core kernel of many parallel codebases, particularly for reductions over a single value per process. The commonly used allreduce recursive-doubling algorithm obtains the lower bound message count, yielding optimality for small reduction sizes based on nodeagnostic performance models. However, this algorithm yields duplicate messages between sets of nodes. Node-aware optimizations in MPICH remove duplicate messages through use of a single master process per node, yielding a large number of inactive processes at each inter-node step. In this paper, we present an algorithm that uses the multiple processes available per node to reduce the maximum number of inter-node messages communicated by a single process, improving the performance of allreduce operations, particularly for small message sizes.

show abstract

Improving Performance Models for Irregular Point-to-Point Communication

Cited by 11 publications

References 20 publications

Performance Analysis and Optimal Node-Aware Communication for Enlarged Conjugate Gradient Methods

Performance Analysis and Optimal Node-Aware Communication for Enlarged Conjugate Gradient Methods

Reducing communication in algebraic multigrid with multi-step node aware communication

Node-Aware Improvements to Allreduce

Contact Info

Product

Resources

About