Matrix Multiplication in Multiword Arithmetic: Error Analysis and Application to GPU Tensor Cores

Abstract

In multiword arithmetic, a matrix is represented as the unevaluated sum of two or more lower-precision matrices, and a matrix product is formed by multiplying the constituents in low precision. We investigate the use of multiword arithmetic for improving the performance-accuracy tradeoff of matrix multiplication with mixed precision block fused multiply-add (FMA) hardware, focusing especially on the tensor cores available on NVIDIA GPUs. Building on a general block FMA framework, we develop a comprehensive error analysis of multiword matrix multiplication. After confirming the theoretical error bounds experimentally by simulating low precision in software, we use the cuBLAS and CUTLASS libraries to implement a number of matrix multiplication algorithms using double-fp16 (double-binary16) arithmetic. When running the algorithms on NVIDIA V100 and A100 GPUs, we find that double-fp16 is not as accurate as fp32 (binary32) arithmetic despite satisfying the same worst-case error bound. Using probabilistic error analysis, we explain why this issue is likely to be caused by the rounding mode used by the NVIDIA tensor cores, and propose a parameterized blocked summation algorithm that alleviates the problem and significantly improves the performance-accuracy tradeoff.

Accurate Calculation of Euclidean Norms Using Double-word Arithmetic

“…It is possible that new operations recently introduced in the IEEE 754 Standard for FP arithmetic [27] and brie y presented in Section 1.3.2 replace these building blocks in a near future. Double-word arithmetic (and more generally, pair and multiple-word arithmetics) is slowly yet steadily gaining importance among numerical methods [32,43,35,17]: this makes a careful study of its error useful.…”

“…The proof is given in the supplementary materials. Let us now compare the bounds of Algorithm 10 and Algorithm 11, i.e., the bounds ( 15) and (17), with the bound of Graillat et al 's algorithm [18] (derived from the relative error bound 3u 2 of Algorithm 5), namely…”

Cascading GEMM: High Precision from Low Precision

“…For instance, [23] expores the use of low precision tensor cores on Nvidia GPUs to do cascading matrices with whatever low precision the tensor core supports. Similarly, in [11] it is also shown how to perform multi-precision GEMM. Both extend the ideas in [13].…”

“…Early work on breaking up higher precision GEMM in terms of lower precision GEMM was given in [13] and extended in [11]. We view present paper as encompassing much of that work, although explained using the example of cascading FP64x2 in terms of FP64 matrix multiplication.…”

“…The advent of processors with new floating point precisions, including various low precision formats (like half-precision), has raised the question of how to leverage the higher performance of low-precision arithmetic into high performance for high-precision arithmetic [1,11,13]. In this paper, we gain insight into possibilities by primarily focusing on one end of the spectrum: the implementation of double-double precision (FP64x2) matrix-matrix multiplication (GEMM) in terms of high-performance double precision (FP64) GEMM.…”