Fixed-rank matrix factorizations and Riemannian low-rank optimization

Abstract

Motivated by the problem of learning a linear regression model whose parameter is a large fixed-rank non-symmetric matrix, we consider the optimization of a smooth cost function defined on the set of fixed-rank matrices. We adopt the geometric framework of optimization on Riemannian quotient manifolds. We study the underlying geometries of several well-known fixed-rank matrix factorizations and then exploit the Riemannian quotient geometry of the search space in the design of a class of gradient descent and trustregion algorithms. The proposed algorithms generalize our previous results on fixed-rank symmetric positive semidefinite matrices, apply to a broad range of applications, scale to high-dimensional problems and confer a geometric basis to recent contributions on the learning of fixed-rank non-symmetric matrices. We make connections with existing algorithms in the context of low-rank matrix completion and discuss relative usefulness of the proposed framework. Numerical experiments suggest that the proposed algorithms compete with the state-of-the-art and that manifold optimization offers an effective and versatile framework for the design of machine learning algorithms that learn a fixed-rank matrix.

Low-rank retractions: a survey and new results

“…Applications of (1) appear in particular in learning problems, where the low-rank constraint is inherent to the model or introduced to reduce memory usage and computation time; see the list of applications in the introduction of [MMBS13b].…”

“…Several other representations exist, see [MMBS13b,§3], but in this paper we will only make use of the three representations above, with an emphasis on (5). Note that the mappings …”

Tangent-Bundle Maps on the Grassmann Manifold: Application to Empirical Arithmetic Averaging

“…A review of the notion, the formal definition and the properties of a class of tangent-bundle maps from a horizontal space to a quotient manifold termed retractions may be found, e.g., in the publicly available report [26]. In the present contribution, a different definition is adopted, which is tailored to the problem of averaging over Grassmannians.…”

Blind demixing for low-latency communication

“…In particular, the factorization M k = w k w H k where w k ∈ C N +K is prevalent in dealing with rank-one Hermitian positive semidefinite matrices [27], [28]. This factorization also takes advantages of lower-dimensional search space [20] over the other general forms of matrix factorization for rankone matrices [26]. However, the factorization M k = w k w H k is not unique because the transformation w k → a k w k leaves the matrix w k w H k unchanged, where a k ∈ SU(1) := {a k ∈ C : a k a * k = a * k a k = 1} and SU(1) is the special unitary group of degree 1.…”