Adaptive regularization with cubics on manifolds

Abstract

Adaptive regularization with cubics (ARC) is an algorithm for unconstrained, nonconvex optimization. Akin to the trust-region method, its iterations can be thought of as approximate, safe-guarded Newton steps. For cost functions with Lipschitz continuous Hessian, ARC has optimal iteration complexity, in the sense that it produces an iterate with gradient smaller than ε in O(1/ε 1.5 ) iterations. For the same price, it can also guarantee a Hessian with smallest eigenvalue larger than − √ ε. In this paper, we study a generalization of ARC to optimization on Riemannian manifolds. In particular, we generalize the iteration complexity results to this richer framework. Our central contribution lies in the identification of appropriate manifold-specific assumptions that allow us to secure these complexity guarantees both when using the exponential map and when using a general retraction. A substantial part of the paper is devoted to studying these assumptions-relevant beyond ARC-and providing user-friendly sufficient conditions for them. Numerical experiments are encouraging. Keywords Optimization on manifolds • Complexity • Lipschitz regularity • Cubic regularization • Newton's method Mathematics Subject Classification 90C26 Nonconvex programming global optimization • 53Z99 Applications of differential geometry to sciences and engineering • 90C53 Methods of quasi-Newton type • 65K05 Numerical mathematical programming methods Authors are listed alphabetically.

Hessian barrier algorithms for non-convex conic optimization

“…In fact, this agrees with the typical restricted Lipschitz gradient condition for the pullback fx (•) = f (x + •) : F x → R, employing the obvious retraction; see [3,17]. ✸ Remark 3.…”

A Decomposition Augmented Lagrangian Method for Low-rank Semidefinite Programming

“…We now state our main complexity result for Algorithm 2.2. The proof follows Theorem 3 in [4], which assumes that the objective function is twice differentiable on M. However, the Riemannian gradient mapping of Ψ is only semismooth. To upper bound grad Ψ(R l+1 ) F , we compare the difference between grad Ψ(R l ) and grad Ψ(R l+1 ) from different tangent space in the Euclidean sense.…”

“…According to Assumption 1.A, it is easy to verify that Ψ is bounded from below by a finite constant Ψ low . The lemma below following from [4] shows that the sum of U l is bounded above by some constant. The proof can be found in Appendix SM2.7.…”

“…According to Lemma 3 in [4], we know that the number K of successful iterations among 0, • • • , l satisfies…”

Modern optimization theory and applications

“…在二阶算法方面, Riemann 信赖域算法是一个常用的方法. 最近, 文献[161] 提出了流 形上的自适应正则化 Newton 算法, 文献[165,166] 将 3 次正则化方法推广到了流形优化中, 文献[167] 对正交约束问题设计了一种结构拟 Newton 算法.…”