Fast Structured Matrix Computations: Tensor Rank and Cohn–Umans Method

Abstract

We discuss a generalization of the Cohn-Umans method, a potent technique developed for studying the bilinear complexity of matrix multiplication by embedding matrices into an appropriate group algebra. We investigate how the Cohn-Umans method may be used for bilinear operations other than matrix multiplication, with algebras other than group algebras, and we relate it to Strassen's tensor rank approach, the traditional framework for investigating bilinear complexity. To demonstrate the utility of the generalized method, we apply it to find the fastest algorithms for forming structured matrix-vector product, the basic operation underlying iterative algorithms for structured matrices. The structures we study include Toeplitz, Hankel, circulant, symmetric, skew-symmetric, f -circulant, block-Toeplitz-Toeplitz-block, triangular Toeplitz matrices, Toeplitzplus-Hankel, sparse/banded/triangular. Except for the case of skew-symmetric matrices, for which we have only upper bounds, the algorithms derived using the generalized Cohn-Umans method in all other instances are the fastest possible in the sense of having minimum bilinear complexity. We also apply this framework to a few other bilinear operations including matrix-matrix, commutator, simultaneous matrix products, and briefly discuss the relation between tensor nuclear norm and numerical stability.

Algorithms for structured matrix-vector product of optimal bilinear complexity

“…For sparse matrices associated with Ω, the matrix-vector product has optimal bilinear complexity #Ω realized by the usual matrix-vector product algorithm [14].…”

“…This article is an addendum to our work in [14] where we proposed a generalization of the Cohn-Umans method [3], [4] and used it to study the bilinear complexity of structured matrix-vector product. We did not derive any actual algorithms in [14]. The purpose of this present work is to provide explicit algorithms for structured matrix-vector product obtained by our generalized Cohn-Umans method in [14].…”

“…We did not derive any actual algorithms in [14]. The purpose of this present work is to provide explicit algorithms for structured matrix-vector product obtained by our generalized Cohn-Umans method in [14]. All algorithms in this paper have been shown to be the fastest possible in terms of bilinear complexity.…”

“…has bilinear complexity one. For those familiar with tensor rank [8], the bilinear complexity of β is just the tensor rank of the structure tensor µ β ∈ C 2 ⊗ C 2 ⊗ C 2 corresponding to β [2], [14].…”

“…, a n−1 ) and b ∈ C is arbitrary. Using this embedding, we obtain Algorithm 2 for Toeplitz matrixvector product [1], [14] . Algorithm 2 Toeplitz matrix-vector product 1: Express the Toeplitz matrix A as (a 1 , .…”