Reproducible BLAS Routines with Tunable Accuracy Using Ozaki Scheme for Many-Core Architectures

Mukunoki, Daichi; Ogita, Takeshi; Ozaki, Katsuhisa

doi:10.1007/978-3-030-43229-4_44

Cited by 14 publications

(19 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, Mukunoki and Ogita presented their approach to implement reproducible BLAS, called OzBLAS [18], with tunable accuracy. This approach is different from both ReproBLAS and ExBLAS as it does not require to implement every BLAS routine from scratch but relies on high-performance (vendor) implementations.…”

Section: Related Workmentioning

confidence: 99%

“…It has led to the inclusion of error-free transformations (EFTs) for addition and multiplication -to return the exact outcome as the result and the error -to assure numerical reproducibility of floating-point operations, into the revised version of the standard. These mechanisms, once implemented in hardware, will simplify our reproducible algorithms -like the ones used in the ExBLAS [6], ReproBLAS [7], OzBLAS [18] libraries -and boost their performance. There are two approaches that enable the addition of floating-point numbers without incurring round-off errors or with reducing their impact.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reproducibility strategies for parallel Preconditioned Conjugate Gradient

Iakymchuk

Barreda

Wiesenberger

et al. 2020

Journal of Computational and Applied Mathematics

View full text Add to dashboard Cite

The Preconditioned Conjugate Gradient method is often used in numerical simulations. While being widely used, the solver is also known for its lack of accuracy while computing the residual. In this article, we aim at a twofold goal: enhance the accuracy of the solver but also ensure its reproducibility in a message-passing implementation. We design and employ various strategies starting from the ExBLAS approach (through preserving every bit of information until final rounding) to its more lightweight performance-oriented variant (through expanding the intermediate precision). These algorithmic strategies are reinforced with programmability suggestions to assure deterministic executions. Finally, we verify these strategies on modern HPC systems: both versions deliver reproducible number of iterations, residuals, direct errors, and vector-solutions for the overhead of only 29 % (ExBLAS) and 4 % (lightweight) on 768 processes.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Reproducibility strategies for parallel Preconditioned Conjugate Gradient

Iakymchuk

Barreda

Wiesenberger

et al. 2020

Journal of Computational and Applied Mathematics

View full text Add to dashboard Cite

show abstract

“…Recently, Mukunoki and Ogita presented their approach to implement reproducible BLAS, called OzBLAS Mukunoki et al (2020), with tunable accuracy. This approach is different from both ReproBLAS and ExBLAS as it does not require to implement every BLAS routine from scratch but relies on high-performance (vendor) implementations.…”

Section: Related Workmentioning

confidence: 99%

“…Ensuring the bit-wise reproducibility is often a complex and expensive task that imposes modifications to the algorithm and its underlying parts such as the BLAS (Basic Linear Algebra Subprograms) routines Lawson et al (1979); Dongarra et al (1990). These modifications are necessary to preserve every bit of information (both result and error) Collange et al (2015) or, alternatively, to cut off some parts of the data and operate on the remaining most significant parts Mukunoki et al (2020); Demmel and Nguyen (2015). Furthermore, the bit-wise reproducibility can become expensive with the overhead of at least 8% for parallel reduction Collange et al (2015); Demmel and Nguyen (2015), up to 2x–4x for matrix-vector product Iakymchuk et al (2019b), and more than 10x for matrix–matrix multiplication Iakymchuk et al (2016).…”

Section: Introductionmentioning

confidence: 99%

“…It has led to the inclusion of error-free transformations (EFTs) for addition and multiplication—to return the exact outcome as the result and the error—to assure numerical reproducibility of floating-point operations, into the revised version of the standard. These mechanisms, once implemented in hardware, will simplify our reproducible algorithms—like the ones used in the ExBLAS Collange et al (2015), ReproBLAS Demmel and Nguyen (2015), OzBLAS Mukunoki et al (2020) libraries—and boost their performance.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reproducibility of parallel preconditioned conjugate gradient in hybrid programming environments

Iakymchuk

Barreda

Graillat³

et al. 2020

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

The Preconditioned Conjugate Gradient method is often employed for the solution of linear systems of equations arising in numerical simulations of physical phenomena. While being widely used, the solver is also known for its lack of accuracy while computing the residual. In this article, we propose two algorithmic solutions that originate from the ExBLAS project to enhance the accuracy of the solver as well as to ensure its reproducibility in a hybrid MPI + OpenMP tasks programming environment. One is based on ExBLAS and preserves every bit of information until the final rounding, while the other relies upon floating-point expansions and, hence, expands the intermediate precision. Instead of converting the entire solver into its ExBLAS-related implementation, we identify those parts that violate reproducibility/non-associativity, secure them, and combine this with the sequential executions. These algorithmic strategies are reinforced with programmability suggestions to assure deterministic executions. Finally, we verify these approaches on two modern HPC systems: both versions deliver reproducible number of iterations, residuals, direct errors, and vector-solutions for the overhead of less than 37.7% on 768 cores.

show abstract

DGEMM Using Tensor Cores, and Its Accurate and Reproducible Versions

Mukunoki

Ozaki

Ogita

et al. 2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

This paper proposes a method for implementing dense matrix multiplication on FP64 (DGEMM) and FP32 (SGEMM) using Tensor Cores on NVIDIA's graphics processing units (GPUs). Tensor Cores are special processing units that perform 4 × 4 matrix multiplications on FP16 inputs with FP32 precision, and return the result on FP32. The proposed method adopts the Ozaki scheme, an accurate matrix multiplication algorithm based on error-free transformation for matrix multiplication. The proposed method has three prominent advantages: first, it can be built upon the cublasGemmEx routine using Tensor Core operations; second, it can achieve higher accuracy than standard DGEMM, including the correctly-rounded result; third, it ensures bit-level reproducibility even for different numbers of cores and threads. The achievable performance of the method depends on the absolute-value range of each element of the input matrices. For example, when the matrices were initialized with random numbers over a dynamic range of 1E+9, our DGEMM-equivalent implementation achieved up to approximately 980 GFlops of FP64 operation on the Titan RTX GPU (with 130 TFlops on Tensor Cores), although cublasDgemm can achieve only 539 GFlops on FP64 floating-point units. Our results reveal the possibility of utilizing hardware with limited FP32/FP64 resources and fast low-precision processing units (such as AI-oriented processors) for general-purpose workloads. Keywords: Tensor cores • FP16 • Half-precision • Low-precision • Matrix multiplication • GEMM • Linear algebra • Accuracy • Reproducibility

show abstract

Reproducible BLAS Routines with Tunable Accuracy Using Ozaki Scheme for Many-Core Architectures

Cited by 14 publications

References 8 publications

Reproducibility strategies for parallel Preconditioned Conjugate Gradient

Reproducibility strategies for parallel Preconditioned Conjugate Gradient

Reproducibility of parallel preconditioned conjugate gradient in hybrid programming environments

DGEMM Using Tensor Cores, and Its Accurate and Reproducible Versions

Contact Info

Product

Resources

About