Low-Rank Optimization with Trace Norm Penalty

Abstract

The paper addresses the problem of low-rank trace norm minimization. We propose an algorithm that alternates between fixed-rank optimization and rank-one updates. The fixed-rank optimization is characterized by an efficient factorization that makes the trace norm differentiable in the search space and the computation of duality gap numerically tractable. The search space is nonlinear but is equipped with a particular Riemannian structure that leads to efficient computations. We present a second-order trust-region algorithm with a guaranteed quadratic rate of convergence. Overall, the proposed optimization scheme converges super-linearly to the global solution while maintaining complexity that is linear in the number of rows and columns of the matrix. To compute a set of solutions efficiently for a grid of regularization parameters we propose a predictorcorrector approach that outperforms the naive warm-restart approach on the fixed-rank quotient manifold. The performance of the proposed algorithm is illustrated on problems of low-rank matrix completion and multivariate linear regression.

Low-rank retractions: a survey and new results

“…Several techniques have been proposed to address (1)-or more specific instances thereof-by exploiting the fact that M r is a submanifold of the Euclidean space R m×n ; see, e.g., [MMBS13a, The set M r is known to be a submanifold of dimension (m+n−r)r embedded in the Euclidean space R m×n [Lee03,Example 8.14]. The low-rank optimization problem (1) is thus in the field of play of Riemannian optimization; see, e.g., [AMS08].…”

Low-rank matrix completion via preconditioned optimization on the Grassmann manifold

“…The authors propose an efficient SVD-based initial guess for U and V which they refine using a Riemannian steepest descent method [2], along with strong theoretical guarantees. Ngo and Saad [33] exploit this idea further by applying a Riemannian conjugate gradient method to this formulation. They endow the Grassmannians with a preconditioned metric in order to better capture the conditioning of low-rank matrix completion, with excellent results.…”

“…Mishra et al [32] propose another geometric approach. They address the problem of low-rank trace norm minimization and propose an algorithm that alternates between fixed-rank optimization and rank-one updates, with applications to low-rank matrix completion.…”

“…Meyer et al [28] and Absil et al [3] propose a few quotient geometries for the manifold of fixed-rank matrices. In recent work, Mishra et al [31] endow these geometries with preconditioned metrics, akin to the ones developed simultaneously by Ngo and Saad [33] on a double Grassmannian, and use Riemannian conjugate gradient methods, also with excellent results. Schneider and Uschmajew [37] propose a recent analysis of the related problem of optimization over the variety of matrices of bounded rank, rather than fixed rank.…”

“…Furthermore, the OptSpace regularization term is efficiently computable since USV F = S F , but it penalizes all entries instead of just the entries (i, j) / ∈ Ω. The preconditioned metrics introduced in [31] and [33] bring notable improvements, suggesting that one should strive for a formulation which admits efficient preconditioners.…”