Randomized algorithms for rounding in the Tensor-Train format

Abstract

The Tensor-Train (TT) format is a highly compact low-rank representation for highdimensional tensors. TT is particularly useful when representing approximations to the solutions of certain types of parametrized partial differential equations. For many of these problems, computing the solution explicitly would require an infeasible amount of memory and computational time. While the TT format makes these problems tractable, iterative techniques for solving the PDEs must be adapted to perform arithmetic while maintaining the implicit structure. The fundamental operation used to maintain feasible memory and computational time is called rounding, which truncates the internal ranks of a tensor already in TT format. We propose several randomized algorithms for this task that are generalizations of randomized low-rank matrix approximation algorithms and provide significant reduction in computation compared to deterministic TT-rounding algorithms. Randomization is particularly effective in the case of rounding a sum of TT-tensors (where we observe 20× speedup), which is the bottleneck computation in the adaptation of GMRES to vectors in TT format. We present the randomized algorithms and compare their empirical accuracy and computational time with deterministic alternatives.

Cost-efficient Gaussian Tensor Network Embeddings for Tensor-structured Inputs

“…Compared to existing sketchingbased ALS algorithm, this algorithm yields better asymptotic computational cost under several regimes, such as when the CP rank is much lower than each dimension size of the input tensor. We also provide analysis on the recently introduced randomized tensor train rounding algorithm [10]. We show that the tensor train embedding used in that algorithm satisfies the accuracy sufficient condition in Section 3 and yields the optimal sketching asymptotic cost, implying that this is an efficient algorithm, and embeddings with other structures cannot achieve lower asymptotic cost.…”

“…While we allow for the data tensor network to be a hypergraph, we consider only graph embeddings, (detailed definition in Section 2), which include tree embeddings that have been previously studied [1,10,4]. Each one of these embeddings consisting of N E tensors can be reduced to a sequence of N E sketches (random sketching matrices).…”

“…Since the tensor train embedding contains N vertices, Corollary 3.2 directly implies that a sketch size of m = Θ N log(1/δ)/ 2 is sufficient for the MPS embedding to be ( , δ)-accurate. This embedding has already been used in applications including tensor train rounding [10] and low rank approximation of matrix product operators [4].…”

“…Note that the tensor train embedding introduced in this work and [10] adds an output sketch dimension to the standard tensor train, and restricts the tensor train rank to be the sketch size m. This is different from the recent work by Rakhshan and Rabusseau [38], where they construct an embedding consisting of m independent tensor trains, each one with a tensor train rank of R. A sketch size upper bound of m = Θ 1/ 2 • (1 + 2/R) N log 2N (1/δ) is derived for that embedding to be ( , δ)-accurate. However, this bound has an exponential dependence on N .…”

“…the optimization problem in (4.1). Tree embeddings, in particular the tensor train embedding, have been widely discussed and used in prior work [38,10,4]. We design an algorithm to sketch with tree embeddings.…”