On the Efficient Implementation of the Matrix Exponentiated Gradient Algorithm for Low-Rank Matrix Optimization

Abstract

Convex optimization over the spectrahedron, i.e., the set of all real n × n positive semidefinite matrices with unit trace, has important applications in machine learning, signal processing and statistics, mainly as a convex relaxation for optimization with low-rank matrices. It is also one of the most prominent examples in the theory of first-order methods for convex optimization in which non-Euclidean methods can be significantly preferable to their Euclidean counterparts, and in particular the Matrix Exponentiated Gradient (MEG) method which is based on the Bregman distance induced by the (negative) von Neumann entropy. Unfortunately, implementing MEG requires a full SVD computation on each iteration, which is not scalable to highdimensional problems.In this work we propose efficient implementations of MEG, both with deterministic and stochastic gradients, which are tailored for optimization with low-rank matrices, and only use a single low-rank SVD computation on each iteration. We also provide efficiently-computable certificates for the correct convergence of our methods. Mainly, we prove that under a strict complementarity condition, the suggested methods converge from a "warm-start" initialization with similar rates to their full-SVD-based counterparts. Finally, we bring empirical experiments which both support our theoretical findings and demonstrate the practical appeal of our methods.

Low-Rank Extragradient Method for Nonsmooth and Low-Rank Matrix Optimization Problems

“…Since, as in the works [17,18] mentioned before which deal with smooth objectives, strict complementarity plays a key role in our analysis, we refer the interested reader to the recent works [16,39,13,20] which also exploit this property for efficient smooth and convex optimization over the spectrahedron. Strict complementarity has also played an instrumental role in two recent and very influential works which used it to prove linear convergence rates for proximal gradient methods [42,14].…”

Low-Rank Mirror-Prox for Nonsmooth and Low-Rank Matrix Optimization Problems

“…In our recent work [21], similar results were obtained for smooth objective functions, when using Non-Eucldiean von Neumann entropy-based gradient methods, a.k.a matrix exponentiated gradient methods [39]. The importance of these methods lies in the fact that they allow to measure the Lipchitz parameters (either of the function or its gradients) with respect to the matrix spectral norm, which can lead to significantly better convergence rates, than when measuring these with respect to the Euclidean norm.…”

“…Computing the matrix logarithm and matrix exponent for the MEG update requires computing a full-rank SVD and returns a full-rank matrix, unlike the truncation of the lower eigenvalues in Euclidean projections. Nevertheless, as shown in [21], under a strict complementarity assumption, and when close to an optimal rank-r solution, it is possible to approximate these steps with updates that only require rank-r SVDs, while sufficiently controlling the errors resulting from the approximations. These approximated steps can be written as…”

“…Extending the results of [18,19,21] to the nonsmooth setting is difficult since the smoothness assumption is crucial to the analysis. Moreover, while [18,19,21] rely on certain eigen-gaps in the gradients at optimal points, for nonsmooth problems, since the subdifferential set is often not a singleton, it is not likely that a similar eigen-gap property holds for all subgradients of an optimal solution.…”

“…In this paper we establish that under the mild assumption that Problem (1) can be formulated as a smooth convex-concave saddle-point problem, i.e., the nonsmooth term can be written as a maximum over (possibly infinite number of) smooth convex functions, we can obtain results in the spirit of [18,19,21]. Concretely, we show that if a strict complementarity (SC) assumption holds for a low-rank optimal solution (see Assumption 1 in the sequel), approximated mirror-prox methods for smooth convex-concave saddlepoint problems (see Algorithm 2 below), which are either based on Euclidean projected gradient updates or MEG updates, and when initialized in the proximity of an optimal solution, converge with their original convergence rate of O(1/t), while requiring only two low-rank SVDs per iteration 2 .…”