Improving the Performance of CA-GMRES on Multicores with Multiple GPUs

Yamazaki, Ichitaro; Anzt, Hartwig; Tomov, Stanimire; Hoemmen, Mark Frederick; Dongarra, Jack

doi:10.1109/ipdps.2014.48

Cited by 45 publications

(35 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, SVQR still requires computing the Gram matrix, leading to the same normwise upper-bound on the orthogonality error as CholQR. In this paper, we focus on CholQR since its implementation is simpler than that of SVQR, and in our previous studies, we did not identify a test case where CA-GMRES converges with SVQR but not with CholQR [25]. Nevertheless, most of the numerical analysis for CholQR can be trivially extended to SVQR [21].…”

Section: Communication-avoiding Qr (Caqr)mentioning

confidence: 99%

“…In addition, Figure 4 shows their performance on up to three NVIDIA Tesla M2090 GPUs. A more detailed description of our implementations and the performance of these standard orthogonalization schemes can be found in [25]. 3.…”

Section: Communication-avoiding Qr (Caqr)mentioning

confidence: 99%

“…To generate n + 1 orthonormal basis vectors, our CA-GMRES and CA-Lanczos [25] use block orthogonalization followed by TSQR with a step size of s, where the step size is equivalent to the block size in the blocked algorithm to orthogonalize n + 1 vectors (e.g., n = 60 and s = 15 in our experiments). We present the experimental results with CA-GMERS and CA-Lanczos in section 4, but we have not studied the blocked algorithms for TSQR.…”

Section: Error Analysismentioning

confidence: 99%

“…Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php numerical studies [25]. However, MGS is based on BLAS-1 and obtains only a fraction of the standard CholQR performance, as shown in Figure 4.…”

Section: Theorem 32 the Orthogonality Error Norm Of The Matrix Q Comentioning

confidence: 99%

“…Our Lanczos implementation uses the three-term recurrence followed by the full reorthogonalization to maintain the orthogonality among the basis vectors, and thick restart [24] to improve its convergence rate. A more detailed description of our implementation can be found in [25].…”

Section: Algorithmsmentioning

confidence: 99%

See 4 more Smart Citations

Mixed-Precision Cholesky QR Factorization and Its Case Studies on Multicore CPU with Multiple GPUs

Yamazaki¹,

Tomov²,

Dongarra³

2015

SIAM J. Sci. Comput.

Self Cite

View full text Add to dashboard Cite

Abstract. To orthonormalize the columns of a dense matrix, the Cholesky QR (CholQR) requires only one global reduction between the parallel processing units and performs most of its computation using BLAS-3 kernels. As a result, compared to other orthogonalization algorithms, CholQR obtains superior performance on many of the current computer architectures, where the communication is becoming increasingly expensive compared to the arithmetic operations. This is especially true when the input matrix is tall-skinny. Unfortunately, the orthogonality error of CholQR depends quadratically on the condition number of the input matrix, and it is numerically unstable when the matrix is ill-conditioned. To enhance the stability of CholQR, we recently used mixed-precision arithmetic; the input and output matrices are in the working precision, but some of its intermediate results are accumulated in the doubled precision. In this paper, we analyze the numerical properties of this mixed-precision CholQR. Our analysis shows that by selectively using the doubled precision, the orthogonality error of the mixed-precision CholQR only depends linearly on the condition number of the input matrix. We provide numerical results to demonstrate the improved numerical stability of the mixed-precision CholQR in practice. We then study its performance. When the target hardware does not support the desired higher precision, software emulation is needed. For example, using software-emulated double-double precision for the working 64-bit double precision, the mixed-precision CholQR requires about 8.5× more floating-point instructions than that required by the standard CholQR. On the other hand, the increase in the communication cost using the double-double precision is less significant, and our performance results on multicore CPU with a different graphics processing unit (GPU) demonstrate that the overhead of using the double-double arithmetic is decreasing on a newer architecture, where the computation is becoming less expensive compared to the communication. As a result, with a latest NVIDIA GPU, the mixed-precision CholQR was only 1.4× slower than the standard CholQR. Finally, we present case studies of using the mixed-precision CholQR within communication-avoiding variants of Krylov subspace projection methods for solving a nonsymmetric linear system of equations and for solving a symmetric eigenvalue problem, on a multicore CPU with multiple GPUs. These case studies demonstrate that by using the higher precision for this small but critical segment of the Krylov methods, we can improve not only the overall numerical stability of the solvers but also, in some cases, their performance.

show abstract

Section: Communication-avoiding Qr (Caqr)mentioning

confidence: 99%

Section: Communication-avoiding Qr (Caqr)mentioning

confidence: 99%

Section: Error Analysismentioning

confidence: 99%

Section: Theorem 32 the Orthogonality Error Norm Of The Matrix Q Comentioning

confidence: 99%

Section: Algorithmsmentioning

confidence: 99%

See 3 more Smart Citations

Mixed-Precision Cholesky QR Factorization and Its Case Studies on Multicore CPU with Multiple GPUs

Yamazaki¹,

Tomov²,

Dongarra³

2015

SIAM J. Sci. Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

Low synchronization Gram–Schmidt and generalized minimal residual algorithms

Swirydowicz

Langou

Ananthan

et al. 2020

Numerical Linear Algebra App

View full text Add to dashboard Cite

Summary The Gram–Schmidt process uses orthogonal projection to construct the A = QR factorization of a matrix. When Q has linearly independent columns, the operator P = I − Q(QTQ)−1QT defines an orthogonal projection onto Q⊥. In finite precision, Q loses orthogonality as the factorization progresses. A family of approximate projections is derived with the form P = I − QTQT, with correction matrix T. When T = (QTQ)−1, and T is triangular, it is postulated that the best achievable orthogonality is 𝒪(ε)κ(A). We present new variants of modified (MGS) and classical Gram–Schmidt algorithms that require one global reduction step. An interesting form of the projector leads to a compact WY representation for MGS. In particular, the inverse compact WY MGS algorithm is equivalent to a lower triangular solve. Our main contribution is to introduce a backward normalization lag into the compact WY representation, resulting in a 𝒪(ε)κ([r0,AVm]) stable Generalized Minimal Residual Method (GMRES) algorithm that requires only one global reduce per iteration. Further improvements in performance are achieved by accelerating GMRES on GPUs.

show abstract