Estimating a Few Extreme Singular Values and Vectors for Large-Scale Matrices in Tensor Train Format

Abstract

We propose new algorithms for singular value decomposition (SVD) of very large-scale matrices based on a low-rank tensor approximation technique called the tensor train (TT) format. The proposed algorithms can compute several dominant singular values and corresponding singular vectors for large-scale structured matrices given in a TT format. The computational complexity of the proposed methods scales logarithmically with the matrix size under the assumption that both the matrix and the singular vectors admit low-rank TT decompositions. The proposed methods, which are called the alternating least squares for SVD (ALS-SVD) and modified alternating least squares for SVD (MALS-SVD), compute the left and right singular vectors approximately through block TT decompositions. The very large-scale optimization problem is reduced to sequential small-scale optimization problems, and each core tensor of the block TT decompositions can be updated by applying any standard optimization methods. The optimal ranks of the block TT decompositions are determined adaptively during iteration process, so that we can achieve high approximation accuracy. Extensive numerical simulations are conducted for several types of TT-structured matrices such as Hilbert matrix, Toeplitz matrix, random matrix with prescribed singular values, and tridiagonal matrix. The simulation results demonstrate the effectiveness of the proposed methods compared with standard SVD algorithms and TT-based algorithms developed for symmetric eigenvalue decomposition.

Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives

“…In particular, we focus on Symmetric Eigenvalue Decomposition (EVD/PCA) and Generalized Eigenvalue Decomposition (GEVD) , Hubig et al, 2015, Huckle and Waldherr, 2012, Kressner et al, 2014a, SVD [Lee and Cichocki, 2015], solutions of overdetermined and undetermined systems of linear algebraic equations [Dolgov andSavostyanov, 2014, Oseledets and, the MoorePenrose pseudo-inverse of structured matrices [Lee and Cichocki, 2016b], and LASSO regression problems [Lee and Cichocki, 2016a]. Tensor networks for extremely large-scale multi-block (multi-view) data are also discussed, especially TN models for orthogonal Canonical Correlation Analysis (CCA) and related Higher-Order Partial Least Squares (HOPLS) problems [Hou, 2017, Hou et al, 2016b, Zhao et al, 2011.…”

“…The traditional methods for solving eigenvalue problems for a symmetric matrix, A P R IˆI , are prohibitive for very large values of I, say I = 10 15 or higher. This computational bottleneck can be very efficiently dealt with through low-rank tensor approximations, and the last 10 years have witnessed the development of such techniques for several classes of optimization problems, including EVD/PCA and SVD , Kressner et al, 2014a, Lee and Cichocki, 2015. The principle is to represent the cost function in a tensor format; under certain conditions, such as that tensors can be often quite well approximated in a low-rank TT format, thus allowing for low-dimensional parametrization.…”

“…In a practical implementation, the full network contraction is never carried out globally, but through iterative sweeps from right to left or vice-versa, to build up L ăn and R ąn from the previous steps. The left and right orthogonalization of the cores can also be exploited to simplify the tensor contraction process , Kressner et al, 2014a, Lee and Cichocki, 2015, Schollwöck, 2011.…”

“…Similarly to the symmetric EVD problem described in the previous section, the block TT concept can be employed to compute only K largest singular values and the corresponding singular vectors of a given matrix A P R IˆJ , by performing the maximization of the following cost function [Cichocki, 2013, Lee and Cichocki, 2015:…”

“…Now, after tensorizing the involved huge-scale matrices, an asymmetric tensor network can be constructed for the computation of K extreme (minimal or maximal) singular values, as illustrated in Figure 3.12 (see [Lee and Cichocki, 2015] for detail and computer simulation experiments). The key idea behind the solutions of (3.40) and (3.41) is to perform TT core contractions to reduce the unfeasible huge-scale optimization problem to relatively small-scale optimization sub-problems, as follows:…”