Conic Geometric Optimization on the Manifold of Positive Definite Matrices

Abstract

We develop geometric optimisation on the manifold of Hermitian positive definite (HPD) matrices. In particular, we consider optimising two types of cost functions: (i) geodesically convex (g-convex); and (ii) log-nonexpansive (LN). G-convex functions are nonconvex in the usual euclidean sense, but convex along the manifold and thus allow global optimisation. LN functions may fail to be even g-convex, but still remain globally optimisable due to their special structure. We develop theoretical tools to recognise and generate g-convex functions as well as cone theoretic fixed-point optimisation algorithms. We illustrate our techniques by applying them to maximum-likelihood parameter estimation for elliptically contoured distributions (a rich class that substantially generalises the multivariate normal distribution). We compare our fixed-point algorithms with sophisticated manifold optimisation methods and obtain notable speedups. 1 To our knowledge the name "geometric optimisation" has not been previously attached to g-convex and cone theoretic HPD matrix optimisation, though several scattered examples do exist. Our theorems offer a formal starting point for recognising HPD geometric optimisation problems. arXiv:1312.1039v3 [math.FA] 12 Dec 2014 2 programming has enjoyed great success across a spectrum of applications-see e.g., the survey of Boyd et al. [11]; we hope this paper helps conic geometric optimisation gain wider exposure.Perhaps the best known conic geometric optimisation problem is computation of the Karcher (Fréchet) mean of a set of HPD matrices, a topic that has attracted great attention within matrix theory [7,8,25,48], computer vision [16], radar imaging [41, Part II], medical imaging [17, 52]-we refer the reader to the recent book [41] for additional applications and references. Another basic geometric optimisation problem arises as a subroutine in image search and matrix clustering [18].Conic geometric optimisation problems also occur in several other areas: statistics (covariance shrinkage) [15], nonlinear matrix equations [31], Markov decision processes and more broadly in the fascinating areas of nonlinear Perron-Frobenius theory [32].As a concrete illustration of our ideas, we discuss the task of maximum likelihood estimate (mle) for elliptically contoured distributions (ECDs) [13,21,37]-see §5. We use ECDs to illustrate our theory, not only because of their instructive value but also because of their importance in a variety of applications [42]. OutlineThe main focus of this paper is on recognising and constructing certain structured nonconvex functions of HPD matrices. In particular, Section 2 studies the class of geodesically convex functions, while Section 4 introduces "log-nonexpansive" functions. We present a limited-memory BFGS algorithm in Section 3, where we also present a derivation for the parallel transport, which, we could not find elsewhere in the literature. Even though manifold optimisation algorithms apply to both classes of functions, for log-nonexpansive functions we advance...

Gradient Method for Optimization on Riemannian Manifolds with Lower Bounded Curvature

“…Consequently, the geometric and algebraic structure that comes from of the Riemannian metric makes possible to greatly reduce the computational cost for solving such problems. Indeed, it is also widely known that, in several contexts, the iteration complexity of the gradient method for convex optimization problems with Lipschitz gradient is much lower than for general nonconvex problems; see for example [6,18,28,33,38] and references therein. Furthermore, many Euclidean optimization problems are naturally posed on the Riemannian context; see [15,18,32,33].…”

“…Indeed, it is also widely known that, in several contexts, the iteration complexity of the gradient method for convex optimization problems with Lipschitz gradient is much lower than for general nonconvex problems; see for example [6,18,28,33,38] and references therein. Furthermore, many Euclidean optimization problems are naturally posed on the Riemannian context; see [15,18,32,33]. Then, to take advantage of the Riemannian geometric structure, it is preferable to treat these problems as the ones of finding singularities of gradient vector fields on Riemannian manifolds rather than using Lagrange multipliers or projection methods; see [23,32,34].…”

“…Accordingly, constrained optimization problems can be viewed as unconstrained ones from a Riemannian geometry point of view. Moreover, Riemannian structures can also opens up new research directions that aid in developing competitive algorithms; see [1,15,18,27,32,33]. For this purpose, extensions of concepts and techniques of optimization from Euclidean space to Riemannian context have been quite frequently in recent years.…”

Iteration-Complexity and Asymptotic Analysis of Steepest Descent Method for Multiobjective Optimization on Riemannian Manifolds

“…Consequently, the geometric and algebraic structures that come from the Riemannian metric make possible to greatly reduce the computational cost for solving such problems. Indeed, it is well known that the iteration-complexity of several optimization methods for convex optimization problems such that objective functions have Lipschitz continuous gradient is much lower than nonconvex optimization problems; see for example [17][18][19][20][21] and references therein. Furthermore, many optimization problems are naturally posed on the Riemannian context; see [18,20,22,23].…”

“…In this sense constrained optimization problems can be seen as unconstrained from the point of view of Riemannian geometry. Moreover, intrinsic Riemannian structures can also opens up new research directions that aid in developing competitive optimization algorithms; see [18,20,22,23,26,27]. More about concepts and techniques of optimization on Riemannian context can be found in [21,25,[28][29][30][31][32][33][34] and the bibliographies therein.…”