CALU: A Communication Optimal LU Factorization Algorithm

Abstract

Abstract. Since the cost of communication (moving data) greatly exceeds the cost of doing arithmetic on current and future computing platforms, we are motivated to devise algorithms that communicate as little as possible, even if they do slightly more arithmetic, and as long as they still get the right answer. This paper is about getting the right answer for such an algorithm.It discusses CALU, a communication avoiding LU factorization algorithm based on a new pivoting strategy, that we refer to as ca-pivoting. The reason to consider CALU is that it does an optimal amount of communication, and asymptotically less than Gaussian elimination with partial pivoting (GEPP), and so will be much faster on platforms where communication is expensive, as shown in previous work.We show that the Schur complement obtained after each step of performing CALU on a matrix A is the same as the Schur complement obtained after performing GEPP on a larger matrix whose entries are the same as the entries of A (sometimes slightly perturbed) and zeros. Hence we expect that CALU will behave as GEPP and it will be also very stable in practice. In addition, extensive experiments on random matrices and a set of special matrices show that CALU is stable in practice. The upper bound on the growth factor of CALU is worse than of GEPP. However, we present examples showing that neither GEPP nor CALU is uniformly more stable than the other.Key words. LU factorization, communication optimal algorithm, numerical stability AMS subject classifications. 65F50, 65F05, 68R101. Introduction. In this paper we discuss CALU, a communication avoiding LU factorization algorithm. The main part of the paper focuses on showing that CALU is stable in practice. We also show that CALU minimizes communication. For this, we use lower bounds on communication for dense LU factorization that were introduced in [5]. These bounds were obtained by showing through reduction that lower bounds on dense matrix multiplication [15,16] represent lower bounds for dense LU factorization as well. These bounds show that a sequential algorithm that computes the LU factorization of a dense n× n matrix transfers between slow and fast memory at least Ω(n 3 /W 1/2 ) number of words and Ω(n 3 /W 3/2 ) number of messages, where W denotes the fast memory size and we assume a message consists of at most W words in consecutive memory locations. On a parallel machine with P processors, if we consider that the local memory size used on each processor is on the order of n 2 /P , so a lower bound on the number of words is Ω(n 2 / √ P ) and a lower bound on the number of messages is Ω( √ P ). Here we consider square matrices, but later we consider the more general case of an m × n matrix.Gaussian elimination with partial pivoting (GEPP) is one of the most stable algorithms for solving a linear system through LU factorization. At each step of the algorithm, the maximum element in each column of L is permuted in diagonal position and used as a pivot. Efficient implementations of this algorithm exist ...

Recursion based parallelization of exact dense linear algebra routines for Gaussian elimination

“…Second, the distant data accesses has an impact on the overall performance. Adapting the communication avoiding techniques of [19] in the framework of our recursive algorithm is highly relevant. Over a finite field, tournament pivoting seem to reduce to computing the union of the non-zero pivots found in concurrent eliminations.…”

Communication-Avoiding Symmetric-Indefinite Factorization

“…When the factorization in Equation (AA4) is computed in a communication-avoiding way using the tall-and-skinny LU factorization [13] (TSLU), L is still bounded, but the bound is 2 bh , where h is a parameter of TSLU that normally satisfies h = O(log n). This can obviously be much larger than 1, although experiments indicate that L is usually much smaller.…”

“…SMLU uses partial pivoting and incurs a slight bandwidth cost overhead compared to RLU (an extra logarithmic factor). Another algorithm which reduces latency cost even further is the communication-avoiding tall-skinny LU algorithm (TSLU) [13]. The algorithm can be applied to general matrices, but the main innovation focuses on tall-skinny matrices.…”

Alan Turing and the origins of modern Gaussian elimination

“…As examples of very recent activities on the error analysis of GE, I discuss here very briefly the research presented in the papers Grigori, Demmel and Xiang (2011) and Dopico and Molera (2012) published in the last two years. Grigori, Demmel and Xiang (2011) consider the LU factorization in the context of one of the hottest topics of numerical computations of the last years: "communication avoiding algorithms".…”

“…Grigori, Demmel and Xiang (2011) consider the LU factorization in the context of one of the hottest topics of numerical computations of the last years: "communication avoiding algorithms". In current and future computers the cost of communication (moving data between different levels of memory or between different processors) greatly exceeds the cost of performing arithmetic operations, therefore there is a strong motivation for developing new algorithms that communicate as little as possible, even if they do more arithmetic.…”