On the Quadratic Convergence of the Cubic Regularization Method under a Local Error Bound Condition

Abstract

In this paper we consider the cubic regularization (CR) method for minimizing a twice continuously differentiable function. While the CR method is widely recognized as a globally convergent variant of Newton's method with superior iteration complexity, existing results on its local quadratic convergence require a stringent non-degeneracy condition. We prove that under a local error bound (EB) condition, which is much weaker a requirement than the existing non-degeneracy condition, the sequence of iterates generated by the CR method converges at least Q-quadratically to a second-order critical point. This indicates that adding a cubic regularization not only equips Newton's method with remarkable global convergence properties but also enables it to converge quadratically even in the presence of degenerate solutions. As a byproduct, we show that without assuming convexity, the proposed EB condition is equivalent to a quadratic growth condition, which could be of independent interest. To demonstrate the usefulness and relevance of our convergence analysis, we focus on two concrete nonconvex optimization problems that arise in phase retrieval and low-rank matrix recovery, respectively, and prove that with overwhelming probability, the sequence of iterates generated by the CR method for solving these two problems converges at least Q-quadratically to a global minimizer. We also present numerical results of the CR method when applied to solve these two problems to support and complement our theoretical development.

On High-Order Multilevel Optimization Strategies

“…Thanks to the use of high-order models, our methods are expected to attain a fast local convergence rate, especially for growing q. The results reported here are inspired by [44] and extend the analysis proposed therein.…”

“…Moreover we establish local convergence results towards second-order stationary points, that are not present neither in [6] nor in [21]. These results not only generalize those in [44], that are valid only for q = 2, but also apply to the one level methods in [6]. From a practical point of view, we implemented the methods of the family corresponding to q = 2 (which represents a multilevel version of the well-known adaptive regularization by cubics) and to q = 3.…”

“…At the same time the proposed analysis also appears as simpler than that in [21], since the regularization parameter λ k is directly updated, rather than the trust-region radius, and since we use the stopping criterion (3.7). Moreover, we also propose local convergence results, which also apply to the methods in [6], and that extend those in [44] to higher-order models.…”

Nonconvex Robust Low-Rank Matrix Recovery

“…By analyzing the optimization geometry, recent works [4,20,30,36,43] have shown that many local search algorithms with either an appropriate initialization or a random initialization can provably solve the low-rank matrix recovery problem (1.2) when the measurement operator A satisfies the RIP. In particular, gradient descent with an appropriate initialization is shown to converge to a global optimum at a linear rate [43,52], while quadratic convergence is established for the cubic regularization method [48]. Key to these results is certain error bound conditions, which elucidate the regularity properties of the underlying optimization problem.…”

A Bregman Forward-Backward Linesearch Algorithm for Nonconvex Composite Optimization: Superlinear Convergence to Nonisolated Local Minima

“…In the convex setting, some studies have shown the potential of variants of regularized Newton [48,81] and semismooth Newton methods [50,82] under a local error bound. In the smooth nonconvex setting, there are many works relying on Levenberg-Marquardt [3,4,36,89], cubic regularization [91], and regularized Newton [85] methods under variants of local error bounds and Hölder metric subregularity.…”