Coordinate Descent Without Coordinates: Tangent Subspace Descent on Riemannian Manifolds

Abstract

Coordinate descent is a well-studied variant of the gradient descent algorithm which offers significant per-iteration computational savings for large-scale problems defined on Euclidean domains. On the other hand, optimization on a general manifold domain can capture a variety of constraints encountered in many problems, but are difficult to model with the usual Euclidean geometry. In this paper, we present an extension of coordinate descent to general manifold domains, and provide a convergence rate analysis for geodesically convex and non-convex smooth objective functions. Our key insight is to draw an analogy between coordinates in Euclidean space and tangent subspaces of a manifold, hence our method is called tangent subspace descent. We study the effect that non-trivial curvature of manifolds have on the convergence rate and its computational properties. We find that on a product manifold, we can achieve per-iteration computational savings analogous to the Euclidean setting. On such product manifolds, we further extend our method to handle non-smooth composite objectives, and provide the corresponding convergence rates.

Riemannian preconditioned coordinate descent for low multi-linear rank approximation

“…Convergence Analysis. The RPCD method can be thought as a variant of Tangent Subspace Descent (TSD) [16]. TSD is the recent generalization of the coordinate descent method to the manifold domain.…”

“…TSD is the recent generalization of the coordinate descent method to the manifold domain. In this section, we generalize the convergence analysis of [16] to the case where exponential map and parallel transport are substituted by retraction and vector transport, respectively. Convergence analysis of the TSD method is a generalization of the Euclidean block coordinate descent method described in [3].…”

“…There are 11 images per subject in different facial expressions or configuration, thus we have a dense tensor X ∈ R 64×64×11×15 . For two levels of compression, we decompose X to three tucker tensor with multi-linear rank (16,16,11,15) and (8,8,11,15), respectively. The results are shown in Figure 2.…”

“…In [16], the authors provided an extension of the coordinate descent method to the manifold domain. The main idea is inexact minimization over subspaces of the tangent space at every point instead of minimization over coordinates (blocks of coordinates).…”

“…• We provide a convergence analysis for the Riemannian coordinate descent algorithm in a relatively general setting. This is done by modification of the convergence analysis in [16] to the case when retraction and vector transport are being used instead of the exponential map and parallel transport.…”