Regularized Computation of Approximate Pseudoinverse of Large Matrices Using Low-Rank Tensor Train Decompositions

Abstract

We propose a new method for low-rank approximation of Moore-Penrose pseudoinverses (MPPs) of large-scale matrices using tensor networks. The computed pseudoinverses can be useful for solving or preconditioning of large-scale overdetermined or underdetermined systems of linear equations. The computation is performed efficiently and stably based on the modified alternating least squares (MALS) scheme using low-rank tensor train (TT) decompositions and tensor network contractions. The formulated large-scale optimization problem is reduced to sequential smaller-scale problems for which any standard and stable algorithms can be applied. Regularization technique is incorporated in order to alleviate ill-posedness and obtain robust low-rank approximations. Numerical simulation results illustrate that the regularized pseudoinverses of a wide class of non-square or nonsymmetric matrices admit good approximate low-rank TT representations. Moreover, we demonstrated that the computational cost of the proposed method is only logarithmic in the matrix size given that the TT-ranks of a data matrix and its approximate pseudoinverse are bounded. It is illustrated that a strongly nonsymmetric convection-diffusion problem can be efficiently solved by using the preconditioners computed by the proposed method.Key words. Alternating least squares (ALS), density matrix renormalization group (DMRG), curse of dimensionality, solving of huge system of linear equations, low-rank tensor approximation, matrix product operators, matrix product states, preconditioning, generalized inverse of huge matrices, tensor networks, big data.

Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions

“…The related optimization problems often involve structured matrices and vectors with over a billion entries (see [67,81,87] and references therein). In particular, we focus on Symmetric Eigenvalue Decomposition (EVD/PCA) and Generalized Eigenvalue Decomposition (GEVD) [70,120,123], SVD [127], solutions of overdetermined and undetermined systems of linear algebraic equations [71,159], the Moore-Penrose pseudo-inverse of structured matrices [129], and Lasso problems [130]. Tensor networks for extremely large-scale multi-block (multiview) data are also discussed, especially TN models for orthogonal Canonical Correlation Analysis (CCA) and related Partial Least Squares (PLS) problems.…”

Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives

“…This chapter introduces feasible solutions for several generic huge-scale dimensionality reduction and related optimization problems, whereby the involved optimized cost functions are approximated by suitable low-rank TT networks. In this way, a very large-scale optimization problem can be converted into a set of much smaller optimization sub-problems of the same kind [Cichocki, 2014, Holtz et al, 2012a, Kressner et al, 2014a, Lee and Cichocki, 2016b, Schollwöck, 2011, which can be solved using standard methods.…”

“…In particular, we focus on Symmetric Eigenvalue Decomposition (EVD/PCA) and Generalized Eigenvalue Decomposition (GEVD) , Hubig et al, 2015, Huckle and Waldherr, 2012, Kressner et al, 2014a, SVD [Lee and Cichocki, 2015], solutions of overdetermined and undetermined systems of linear algebraic equations [Dolgov andSavostyanov, 2014, Oseledets and, the MoorePenrose pseudo-inverse of structured matrices [Lee and Cichocki, 2016b], and LASSO regression problems [Lee and Cichocki, 2016a]. Tensor networks for extremely large-scale multi-block (multi-view) data are also discussed, especially TN models for orthogonal Canonical Correlation Analysis (CCA) and related Higher-Order Partial Least Squares (HOPLS) problems [Hou, 2017, Hou et al, 2016b, Zhao et al, 2011.…”

“…The objective is to find an approximative solution, X P R JˆK , in a TT format, by imposing additional constraints (e.g., via Tikhonov regularization). In a special case when B = I I , for K = I the problem is equivalent to the computation of Moore-Penrose pseudo inverse in an appropriate TT format (see [Lee and Cichocki, 2016b] for computer simulation experiments). Figure 3.19: TT sub-networks for the minimization of a very large-scale regularized cost function J(X) = }AX´B} 2 F + γ}LX} 2 F = tr(X T A T AX)2 tr(B T AX) + γ tr(X T L T LX) + tr(B T B).…”

“…• Solution of huge linear systems of equations, together with the inverse/pseudoinverse of matrices and related problems [Bolten et al, 2016, Lee and Cichocki, 2016b,…”