Nonconvex Factorization and Manifold Formulations are Almost Equivalent in Low-rank Matrix Optimization

Abstract

In this paper, we consider the geometric landscape connection of the widely studied manifold and factorization formulations in low-rank positive semidefinite (PSD) and general matrix optimization. We establish an equivalence on the set of first-order stationary points (FOSPs) and second-order stationary points (SOSPs) between the manifold and the factorization formulations. We further give a sandwich inequality on the spectrum of Riemannian and Euclidean Hessians at FOSPs, which can be used to transfer more geometric properties from one formulation to another. Similarities and differences on the landscape connection under the PSD case and the general case are discussed. To the best of our knowledge, this is the first geometric landscape connection between the manifold and the factorization formulations for handling rank constraints. In the general low-rank matrix optimization, the landscape connection of two factorization formulations (unregularized and regularized ones) is also provided. By applying these geometric landscape connections, we are able to solve unanswered questions in literature and establish stronger results in the applications on geometric analysis of phase retrieval, wellconditioned low-rank matrix optimization, and the role of regularization in factorization arising from machine learning and signal processing.

A Unified Complexity Metric for Nonconvex Matrix Completion and Matrix Sensing in the Rank-one Case

“…Although the convex relaxation is usually guaranteed to recover the exact ground truth with almost the optimal sample complexity, the associated algorithms operate in the space of matrix variables and, thus, are computationally inefficient for largescale problems [60]. Similar issues are observed for algorithms based on the Singular Value Projection [31] and Riemannian optimization algorithms [53,54,30,2,37]. The analysis of the convex relaxation approach in the noisy case is recently conducted by bridging the convex and the nonconvex approaches [21,22].…”

On Geometric Connections of Embedded and Quotient Geometries in Riemannian Fixed-rank Matrix Optimization

“…They showed while the sets of FOSPs under the factorization formulation can be larger, the sets of SOSPs are contained in the set of fixed points of the PGD with a small stepsize. Another related work is [LLZ21], where they studied the landscape connections of the factorization formulation and the Riemannian formulation with embedded geometry for both (1) and (2) and identified the close connections between the landscapes under these two formulations, e.g., the set of FOSPs and SOSPs under these two formulations are exactly the same when constraining to rank-r matrices. In addition, under the quotient geometry, a general equivalence relation on the sets of FOSPs and SOSPs of objectives on the total space and the quotient space is given in [Bou20,Section 9.11].…”

“…As we have discussed in the Introduction, another popular approach for handling the rank constraint in (1) or (2) is via factorizing X into YY J or LR J and then treat the new problem as unconstrained optimization in the Euclidean space. In the recent work [LLZ21], they showed for both (1) and (2) the geometric landscapes under the factorization and embedded submanifold formulations are almost equivalent. By combining their results and the results in this paper, we also have a geometric landscape connection of (1) and (2) under the factorization and quotient manifold formulations.…”