Communication-optimal Parallel and Sequential QR and LU Factorizations

Abstract

We present parallel and sequential dense QR factorization algorithms that are both optimal (up to polylogarithmic factors) in the amount of communication they perform, and just as stable as Householder QR. Our first algorithm, Tall Skinny QR (TSQR), factors m × n matrices in a one-dimensional (1-D) block cyclic row layout, and is optimized for m n. Our second algorithm, CAQR (Communication-Avoiding QR), factors general rectangular matrices distributed in a two-dimensional block cyclic layout. It invokes TSQR for each block column factorization.The new algorithms are superior in both theory and practice. We have extended known lower bounds on communication for sequential and parallel matrix multiplication to provide latency lower bounds, and show these bounds apply to the LU and QR decompositions. We not only show that our QR algorithms attain these lower bounds (up to polylogarithmic factors), but that existing LAPACK and ScaLAPACK algorithms perform asymptotically more communication. We also point out recent LU algorithms in the literature that attain at least some of these lower bounds.Both TSQR and CAQR have asymptotically lower latency cost in the parallel case, and asymptotically lower latency and bandwidth costs in the sequential case. In practice, we have implemented parallel TSQR on several machines, with speedups of up to 6.7× on 16 processors of a Pentium III cluster, and up to 4× on 32 processors of a BlueGene/L. We have also implemented sequential TSQR on a laptop for matrices that do not fit in DRAM, so that slow memory is disk. Our out-of-DRAM implementation was as little as 2× slower than the predicted runtime as though DRAM were infinite.We have also modeled the performance of our parallel CAQR algorithm, yielding predicted speedups over ScaLAPACK's PDGEQRF of up to 9.7× on an IBM Power5, up to 22.9× on a model Petascale machine, and up to 5.3× on a model of the Grid.

A blocked QR-decomposition for the parallel symmetric eigenvalue problem

“…However, it is not numerically stable if A is ill conditioned [1]. A good comparison regarding parallel performance and numerical stability of the cited algorithms can be found in [9]. Table 1 summarizes this work.…”

“…Except of lower order terms, the first step Parallel algorithm Flops on critical path Messages Comm. volume PDGEQRF 2mn 2 P + n 2 2 log(P ) 2n log(P ) n 2 2 log(P ) TSQR 2mn 2 P + 2 3 n 3 log(P ) log(P ) n 2 2 log(P ) CholeskyQR 2mn 2 P + n 3 3 log(P ) log(P ) n 2 2 log(P ) Table 1: Out of [9]: Performance model of selected parallel QR-decomposition algorithms. n × m stands for the size of the matrix, P for the number of parallel processes.…”

“…In [8,9] Demmel et al propose an algorithm named Tall Skinny QR (TSQR). The approach consists of first performing a local QR-decomposition on each process (assuming a 1D parallel data layout).…”

QCG-OMPI: MPI applications on grids

“…We consider a variant of this algorithm, so-called "Communication-Avoiding QR" (CAQR) [14]. As in ScaLAPACK, the basic operation of CAQR is the factorization of a panel followed by the update of the trailing submatrix.…”

“…TSQR is particularly well adapted to the factorization of tall and skinny matrices, i.e., matrices satisfying M ≫ N (see [14]). TSQR is an important kernel for two reasons.…”