Scaling and Scalability: Provable Nonconvex Low-Rank Tensor Estimation from Incomplete Measurements

Abstract

Tensors, which provide a powerful and flexible model for representing multi-attribute data and multiway interactions, play an indispensable role in modern data science across various fields in science and engineering. A fundamental task is to faithfully recover the tensor from highly incomplete measurements in a statistically and computationally efficient manner. Harnessing the low-rank structure of tensors in the Tucker decomposition, this paper develops a scaled gradient descent (ScaledGD) algorithm to directly recover the tensor factors with tailored spectral initializations, and shows that it provably converges at a linear rate independent of the condition number of the ground truth tensor for two canonical problemstensor completion and tensor regression -as soon as the sample size is above the order of n 3/2 ignoring other parameter dependencies, where n is the dimension of the tensor. This leads to an extremely scalable approach to low-rank tensor estimation compared with prior art, which suffers from at least one of the following drawbacks: extreme sensitivity to ill-conditioning, high per-iteration costs in terms of memory and computation, or poor sample complexity guarantees. To the best of our knowledge, ScaledGD is the first algorithm that achieves near-optimal statistical and computational complexities simultaneously for low-rank tensor completion with the Tucker decomposition. Our algorithm highlights the power of appropriate preconditioning in accelerating nonconvex statistical estimation, where the iteration-varying preconditioners promote desirable invariance properties of the trajectory with respect to the underlying symmetry in low-rank tensor factorization.

Provable Tensor-Train Format Tensor Completion by Riemannian Optimization

“…The contract rate is free of the tensor condition number, improving over prior works (Jain and Oh, 2014;Cai et al, 2021a;Xia and Yuan, 2019) the iteration complexity for completing an ill-conditioned tensor. The attained iteration complexity matches the best one achieved by a recently proposed scaled gradient descent algorithm (Tong et al, 2021). Finally, inspired by the idea in Xia and Yuan (2019), we propose a novel initialization, based on a sequential second order moment method, as the input of RGrad algorithm for tensor completion.…”

“…The recursive nature of computing a TT decomposition substantially complicates the theoretical analysis, involving repeated appearances of the incoherence parameters and TT rank. Interestingly, the required number of iterations l max is free of the condition number, which often appears in decomposition-based algorithms (Cai et al, 2021a;Han et al, 2020), except the recently proposed scaled gradient descent algorithm (Tong et al, 2021).…”

“…Later, Xia and Yuan (2019) investigated a polynomial time algorithm for exact tensor completion with a sample complexity O(κ 2 0 r m d m/2 • Polylog(d))) via gradient descent on the Grassmannian manifold, but the iteration complexity was not provided. More recently, Tong et al (2021) introduced a scaled gradient descent algorithm and proved the iteration complexity that is free of the condition number. In the work of , a fast higher-order orthogonal iteration algorithm was proposed for noisy tensor completion achieving a statistically optimal rate with a sample complexity O(r m/2 d m/2 • Polylog(d)).…”

Guaranteed Functional Tensor Singular Value Decomposition

“…It is also worth mentioning that there are several recent results that directly solve the tabular tensor decomposition by (accelerated) gradient method with convergence guarantee (Cai et al, 2019;Han et al, 2021;Tong et al, 2021). Such methods usually have an advantage over power iteration as they directly aim at the optimization of likelihood, and can successfully remove the effect of incoherence from the final estimation error bound.…”

“…Motivated by real data applications in practice, related statistical models were further studied under non-Gaussian or missing-data scenarios. For example, Hong et al (2020); Han et al (2021) considered the likelihood-based generalized low-rank tensor decomposition under general exponential family; Cai et al (2019); Tong et al (2021) studied the low-rank tensor estimation under incomplete measurements.…”