Faster Johnson–Lindenstrauss transforms via Kronecker products

Abstract

The Kronecker product is an important matrix operation with a wide range of applications in signal processing, graph theory, quantum computing and deep learning. In this work, we introduce a generalization of the fast Johnson–Lindenstrauss projection for embedding vectors with Kronecker product structure, the Kronecker fast Johnson–Lindenstrauss transform (KFJLT). The KFJLT reduces the embedding cost by an exponential factor of the standard fast Johnson–Lindenstrauss transform’s cost when applied to vectors with Kronecker structure, by avoiding explicitly forming the full Kronecker products. We prove that this computational gain comes with only a small price in embedding power: consider a finite set of $p$ points in a tensor product of $d$ constituent Euclidean spaces $\bigotimes _{k=d}^{1}{\mathbb{R}}^{n_k}$, and let $N = \prod _{k=1}^{d}n_k$. With high probability, a random KFJLT matrix of dimension $m \times N$ embeds the set of points up to multiplicative distortion $(1\pm \varepsilon )$ provided $m \gtrsim \varepsilon ^{-2} \, \log ^{2d - 1} (p) \, \log N$. We conclude by describing a direct application of the KFJLT to the efficient solution of large-scale Kronecker-structured least squares problems for fitting the CP tensor decomposition.

Tensor Methods for Nonlinear Matrix Completion

“…An alternative approach would be to incorporate matrix sketching techniques to allow for more efficient computation of the SVD in the tensorized domain. For example, recent work that focuses on sketches for matrices with Kronecker product structure, such as TensorSketch [30] and its modifications [1] or the Kronecker fast Johnson--Lindenstrauss transform [21], could potentially be adapted to our setting.…”

Randomized numerical linear algebra: Foundations and algorithms

“…They were first analyzed by Rudelson (2012). The paper (Sun, Guo, Tropp and Udell 2018) proposed the application of tensor random embeddings for randomized linear algebra; some extensions appear in (Jin, Kolda and Ward 2019) and (Malik and Becker 2019). See (Baldi and Vershynin 2019) and (Vershynin 2019) for related theoretical results.…”

Modewise Operators, the Tensor Restricted Isometry Property, and Low-Rank Tensor Recovery

“…More general n-stage modewise operators can be defined similarly. First analyzed in [21,23] for aiding in the rapid computation of the CP decomposition, such modewise compression operators offer a wide variety of computational advantages over standard vector-based approaches (in which R 1 is a vectorization operator so that d " 1, A 1 " A P R mˆś d j n j is a standard Johnson-Lindenstrauss map, and all remaining operators R 2 , B 1 , . .…”

“…In particular, when R 1 is a more modest reshaping (or even the identity) the resulting modewise linear transforms can be formed using significantly fewer random variables (effectively, independent random bits), and stored using less memory by avoiding the use of a single massive mˆś d j n j matrix. In addition, such modewise linear operators also offer trivially parallelizable operations, faster serial data evaluations than standard vectorized approaches do for structured data (see, e.g., [23]), and the ability to better respect the multimodal structure of the given tensor data.…”

“…Other recent work involving the analysis of modewise maps for tensor data include, e.g., applications in kernel learning methods which effectively use modewise operators specialized to finite sets of rank-one tensors [2], as well as a variety of works in the computer science literature aimed at compressing finite sets of low-rank (with respect to, e.g., CP and tensor train decompositions [32]) tensors. More general results involving extensions of bounded orthonormal sampling results to the tensor setting [23,3] apply to finite sets of arbitrary tensors. With respect to norm-preserving modewise embeddings of infinite sets, prior work has been limited to oblivious subspace embeddings (see, e.g., [21,28]).…”