Improving the Numerical Stability of Fast Matrix Multiplication

Ballard, Grey; Benson, Austin R.; Druinsky, Alex; Lipshitz, Benjamin; Schwartz, Oded

doi:10.1137/15m1032168

Cited by 31 publications

(27 citation statements)

References 25 publications

(133 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From (2.12) and Figure 2.10, the equivalent matrix form is C (n) = BA (n) , which allows us to employ established fast matrix-by-vector and matrix-by-matrix multiplications when dealing with very large-scale tensors. Efficient and optimized algorithms for TTM are, however, still emerging [11,12,131].…”

Section: Symmetric Tensor Decompositionmentioning

confidence: 99%

See 1 more Smart Citation

Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions

Cichocki

Lee

Oseledets

et al. 2016

FNT in Machine Learning

371

342

View full text Add to dashboard Cite

Modern applications in engineering and data science are increasingly based on multidimensional data of exceedingly high volume, variety, and structural richness. However, standard machine learning algorithms typically scale exponentially with data volume and complexity of cross-modal couplings - the so called curse of dimensionality - which is prohibitive to the analysis of large-scale, multi-modal and multi-relational datasets. Given that such data are often efficiently represented as multiway arrays or tensors, it is therefore timely and valuable for the multidisciplinary machine learning and data analytic communities to review low-rank tensor decompositions and tensor networks as emerging tools for dimensionality reduction and large scale optimization problems. Our particular emphasis is on elucidating that, by virtue of the underlying low-rank approximations, tensor networks have the ability to alleviate the curse of dimensionality in a number of applied areas. In Part 1 of this monograph we provide innovative solutions to low-rank tensor network decompositions and easy to interpret graphical representations of the mathematical operations on tensor networks. Such a conceptual insight allows for seamless migration of ideas from the flat-view matrices to tensor network operations and vice versa, and provides a platform for further developments, practical applications, and non-Euclidean extensions. It also permits the introduction of various tensor network operations without an explicit notion of mathematical expressions, which may be beneficial for many research communities that do not directly rely on multilinear algebra. Our focus is on the Tucker and tensor train (TT) decompositions and their extensions, and on demonstrating the ability of tensor networks to provide linearly or even super-linearly (e.g., logarithmically) scalable solutions, as illustrated in detail in Part 2 of this monograph

show abstract

Section: Symmetric Tensor Decompositionmentioning

confidence: 99%

“…Efficient and parallel (state of the art) algorithms for multiplications of such very largescale matrices are proposed in[11,131].…”

mentioning

confidence: 99%

Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions

Cichocki

Lee

Oseledets

et al. 2016

FNT in Machine Learning

371

342

View full text Add to dashboard Cite

show abstract

“…Only a few levels of the recursion are exploited in practice because the cost of extra additions and extra memory movements quickly offsets the reduction in floating point operations. Also, Strassen is known to become more numerically unstable particularly when more than two levels of recursion are employed [8,9,10].…”

Section: High-performance Implementations Of Strassenmentioning

confidence: 99%

Generating Families of Practical Fast Matrix Multiplication Algorithms

Huang

Rice

Matthews

et al. 2017

2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

Matrix multiplication (GEMM) is a core operation to numerous scientific applications. Traditional implementations of Strassen-like fast matrix multiplication (FMM) algorithms often do not perform well except for very large matrix sizes, due to the increased cost of memory movement, which is particularly noticeable for non-square matrices. Such implementations also require considerable workspace and modifications to the standard BLAS interface. We propose a code generator framework to automatically implement a large family of FMM algorithms suitable for multiplications of arbitrary matrix sizes and shapes. By representing FMM with a triple of matrices U, V, W that capture the linear combinations of submatrices that are formed, we can use the Kronecker product to define a multi-level representation of Strassen-like algorithms. Incorporating the matrix additions that must be performed for Strassen-like algorithms into the inherent packing and micro-kernel operations inside GEMM avoids extra workspace and reduces the cost of memory movement. Adopting the same loop structures as high-performance GEMM implementations allows parallelization of all FMM algorithms with simple but efficient data parallelism without the overhead of task parallelism. We present a simple performance model for general FMM algorithms and compare actual performance of 20+ FMM algorithms to modeled predictions. Our implementations demonstrate a performance benefit over conventional GEMM on single core and multicore systems. This study shows that Strassen-like fast matrix multiplication can be incorporated into libraries for practical use.

show abstract

“…Sparsity will be an essential ingredient of our analysis, and we better motivate it in Section 4.3. Others [2,3] consider sparsity in analyzing and improving the numerical stability of fast matrix multiplication algorithms, though they do not refer to it by the same name. We use the following notation in proofs of circuit quality.…”

Section: Problem Statementmentioning

confidence: 99%

Constant-Depth and Subcubic-Size Threshold Circuits for Matrix Multiplication

Parekh

Phillips

James

et al. 2018

Proceedings of the 30th on Symposium on Parallelism in Algorithms and Architectures

View full text Add to dashboard Cite

Boolean circuits of McCulloch-Pitts threshold gates are a classic model of neural computation studied heavily in the late 20th century as a model of general computation. Recent advances in large-scale neural computing hardware has made their practical implementation a near-term possibility. We describe a theoretical approach for multiplying two N by N matrices that integrates threshold gate logic with conventional fast matrix multiplication algorithms, that perform O(N ω ) arithmetic operations for a positive constant ω < 3. Our approach converts such a fast matrix multiplication algorithm into a constant-depth threshold circuit with approximately O(N ω ) gates. Prior to our work, it was not known whether the Θ(N 3 )-gate barrier for matrix multiplication was surmountable by constant-depth threshold circuits.Dense matrix multiplication is a core operation in convolutional neural network training. Performing this work on a neural architecture instead of off-loading it to a GPU may be an appealing option.

show abstract

Improving the Numerical Stability of Fast Matrix Multiplication

Cited by 31 publications

References 25 publications

Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions

Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions

Generating Families of Practical Fast Matrix Multiplication Algorithms

Constant-Depth and Subcubic-Size Threshold Circuits for Matrix Multiplication

Contact Info

Product

Resources

About