Computationally Efficient and Statistically Optimal Robust Low-rank Matrix and Tensor Estimation

Abstract

Low-rank matrix estimation under heavy-tailed noise is challenging, both computationally and statistically. Convex approaches have been proven statistically optimal but suffer from high computational costs, especially since robust loss functions are usually non-smooth. More recently, computationally fast non-convex approaches via sub-gradient descent are proposed, which, unfortunately, fail to deliver a statistically consistent estimator even under sub-Gaussian noise. In this paper, we introduce a novel Riemannian sub-gradient (RsGrad) algorithm which is not only computationally efficient with linear convergence but also is statistically optimal, be the noise Gaussian or heavy-tailed with a finite 1 + ε moment. Convergence theory is established for a general framework and specific applications to absolute loss, Huber loss and quantile loss are investigated. Compared with existing non-convex methods, ours reveals a surprising phenomenon of dual-phase convergence. In phase one, RsGrad behaves as in a typical non-smooth optimization that requires gradually decaying stepsizes. However, phase one only delivers a statistically sub-optimal estimator which is already observed in existing literature. Interestingly, during phase two, RsGrad converges linearly as if minimizing a smooth and strongly convex objective function and thus a constant stepsize suffices. Underlying the phase-two convergence is the smoothing effect of random noise to the non-smooth robust losses in an area close but not too close to the truth. Numerical simulations confirm our theoretical discovery and showcase the superiority of RsGrad over prior methods.

Robust low-rank tensor regression via truncation and adaptive Huber loss

“…The Huber loss has been widely used in various regression models such as Fan et al (2017) [3] and Sun et al (2020) [8] for linear regression, Tan et al (2022) [9] for reduced rank regression and Shen et al (2022) [7] for matrix recovery.…”

Provable sample-efficient sparse phase retrieval initialized by truncated power method

“…It is unlikely that we can remove this additional term. Indeed, similar phenomenon exists ubiquitously non-convex algorithms in the literature, such as low rank matrix recovery [44,52] or low rank tensor recovery [12,41,45]. For example, in the matrix case, we can analogously define the stable rank of a rank-r matrix M ∈ R n×n as sr(M)…”

“…where in the last inequality we used (41). The above inequality also implies dist(x 0 , w 0 t ) ⩽ 1.…”

Error Bound of Empirical $\ell_2$ Risk Minimization for Noisy Standard and Generalized Phase Retrieval Problems

“…To enhance robustness, we truncate overlarge responses to robustify the quadratic loss. Note that compared with previous works [12,18,21,46,53,59] that revolve around highdimensional structured signal recovery and convex programming, our problem setting is new and significantly different. To be specific, the objective is non-convex and x 0 is unstructured.…”

“…To address the issue, many techniques in statistical estimation have been developed, especially in mean estimation, including median of means [25,43] and Catoni's mean estimator via robust empirical loss [4,11], also see the survey [38]. Particularly, a shrinkage principle for low-rank trace regression was recently developed in [18], which was then applied in corrupted generalized linear regression [59], matrix and tensor completion [46], one-bit estimation [12], vector autoregressive model [53].…”