Generating Families of Practical Fast Matrix Multiplication Algorithms

Abstract

Matrix multiplication (GEMM) is a core operation to numerous scientific applications. Traditional implementations of Strassen-like fast matrix multiplication (FMM) algorithms often do not perform well except for very large matrix sizes, due to the increased cost of memory movement, which is particularly noticeable for non-square matrices. Such implementations also require considerable workspace and modifications to the standard BLAS interface. We propose a code generator framework to automatically implement a large family of FMM algorithms suitable for multiplications of arbitrary matrix sizes and shapes. By representing FMM with a triple of matrices U, V, W that capture the linear combinations of submatrices that are formed, we can use the Kronecker product to define a multi-level representation of Strassen-like algorithms. Incorporating the matrix additions that must be performed for Strassen-like algorithms into the inherent packing and micro-kernel operations inside GEMM avoids extra workspace and reduces the cost of memory movement. Adopting the same loop structures as high-performance GEMM implementations allows parallelization of all FMM algorithms with simple but efficient data parallelism without the overhead of task parallelism. We present a simple performance model for general FMM algorithms and compare actual performance of 20+ FMM algorithms to modeled predictions. Our implementations demonstrate a performance benefit over conventional GEMM on single core and multicore systems. This study shows that Strassen-like fast matrix multiplication can be incorporated into libraries for practical use.

Strassen's Algorithm for Tensor Contraction

“…• A number of recent papers explore practical implementations of Strassen-like fast matrix multiplications [9,8]. How to extend fast matrix multiplication with different partition block sizes for tensor contraction is an open question.…”

“…As demonstrated in [7], this approach makes Strassen practical for smaller matrices and matrices of special shape (importantly, for rank-k updates, where N p is relatively small comparing to N i and N j ). This research is pushed further [8] by revealing that Strassen performs relatively better than most other Strassen-like FMM algorithms with one or two levels of recursions, when modeled as well as in practice. For this reason, we do not extend those FMM algorithms to TC in this paper, although it may be worthwhile in future work to pursue certain of these algorithms for highly non-square tensor contraction shapes.…”

“…A more comprehensive parallelization scheme, for example using taskbased parallelism [9], may show better performance. Additionally, since the DF/CCSD(T) contractions are highly "non-square", an alternate fast matrix multiplication algorithm [9,8] may perform better. Shape-dependence experiments.…”

“…Standing on the shoulders of giants. This paper builds upon a number of recent developments: The GotoBLAS algorithm for matrix multiplication (GEMM) [1] that underlies the currently fastest implementations of GEMM for CPUs; The refactoring of the GotoBLAS algorithm as part of the BLAS-like Library Instantiation Software (BLIS) [2,3], which exposes primitives for implementing BLAS-like operations; The systematic parallelization of the loops that BLIS exposes so that high-performance can be flexibly attained on multicore and many-core architectures [4]; The casting of tensor contraction (TC) in terms of the BLIS primitives [5,6] without requiring the transposition (permutation) used by traditional implementations; The practical high-performance implementation of the classical Strassen's algorithm (Strassen) [7] in terms of variants of the BLIS primitives; and the extension of this implementation [8] to a family of Strassen-like algorithms (Fast Matrix Multiplication algorithms) [9]. Together, these results facilitate what we believe to be the first extension of Strassen's algorithm to TC.…”

Engineering a Delegatable and Error-Tolerant Algorithm for Counting Small Subgraphs

“…Ballard, Demmel, Holtz, Lipshitz, and Schwartz [15], Ballard, Demmel, Holtz, and Schwartz [16], and Lipshitz, Ballard, Demmel, and Schwartz [77]. Recent engineering work includes Benson and Ballard [23] and Huang, Rice, Matthews, and van de Geijn [65]. Our work differs from these works in that we seek a self-contained proof-of-concept demonstration requiring good-performance finite-field matrix multiplication on GPUs, but we do not necessarily seek the most optimized possible implementation; such optimizations are left for future work.…”