Avoiding communication in sparse matrix computations

Abstract

The performance of sparse iterative solvers is typically limited by sparse matrix-vector multiplication, which is itself limited by memory system and network performance. As the gap between computation and communication speed continues to widen, these traditional sparse methods will suffer. In this paper we focus on an alternative building block for sparse iterative solvers, the "matrix powers ker-, and show that by organizing computations around this kernel, we can achieve nearminimal communication costs. We consider communication very broadly as both network communication in parallel code and memory hierarchy access in sequential code. In particular, we introduce a parallel algorithm for which the number of messages (total latency cost) is independent of the power k, and a sequential algorithm, that reduces both the number and volume of accesses, so that it is independent of k in both latency and bandwidth costs. This is part of a larger project to develop "communication-avoiding Krylov subspace methods," which also addresses the numerical issues associated with these methods. Our algorithms work for general sparse matrices that "partition well".We introduce parallel performance models of matrices arising from 2D and 3D problems and show predicted speedups over a conventional algorithm of up to 7x on a Petaflop-scale machine and up to 22x on computation across the Grid. Analogous sequential performance models of the same problems predict speedups over a conventional algorithm of up to 10x on an out-of-core implementation, and up to 2.5x when we use our ideas to reduce offchip latency and bandwidth to DRAM. Finally, we validate the model on an out-of-core sequential implementation and measured a speedup of over 3x, which is close to the predicted speedup.

An approach for code generation in the Sparse Polyhedral Framework

“…Unstructured cache blocking [20] is another approach. The communication-avoiding algorithms of Demmel et al [18] also create sparse tiles, but those tiles overlap so as to enable parallel execution of the tiles and minimal communication between tiles.…”

ClouDiA: a deployment advisor for public clouds

“…ClouDiA works for two large classes of data-driven applications. The first class, which contains many HPC applications, is sensitive to the worst-link latency, as this latency can significantly affect total time-to-solution in a variety of scientific applications [1,12,22,35]. The second class, represented by search engines as well as web services and portals, is sensitive to the longest path between application nodes, as this cost models the network links with the highest potential impact on application response time.…”

“…As the end of a time step is a logical barrier, worst-link latency essentially determines communication cost [1,12,35,72]. Similar communication patterns are common in multiple linear algebra computations [22]. Another example of an HPC application where time-to-solution is critical is dynamic traffic assignment [64].…”

Resisting Skew-Accumulation for Time-Stepped Applications in the Cloud via Exploiting Parallelism

“…Some studies [31] proposed to avoid communication at the expense of performing some redundant computation. While communicating less often certainly helps, this technique alone cannot deal with latency spikes.…”