Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity

Abstract

An Adaptive Regularisation framework using Cubics (ARC) was proposed for unconstrained optimization and analysed in Cartis, Gould & Toint (Part I, 2007). In this companion paper, we further the analysis by providing worst-case global iteration complexity bounds for ARC and a second-order variant to achieve approximate first-order, and for the latter even second-order, criticality of the iterates. In particular, the second-order ARC algorithm requires at most O(ǫ −3/2) iterations to drive the objective's gradient below the desired accuracy ǫ, and O(ǫ −3), to reach approximate nonnegative curvature in a subspace. The orders of these bounds match those proved by Nesterov & Polyak (Math. Programming 108(1), 2006, pp 177-205) for their Algorithm 3.3 which minimizes the cubic model globally on each iteration. Our approach is more general, and relevant to practical (large-scale) calculations , as ARC allows the cubic model to be solved only approximately and may employ approximate Hessians.

Second-Order Optimality and Beyond: Characterization and Evaluation Complexity in Convexly Constrained Nonlinear Optimization

“…Observe that, because of (4.2) and (4.4), ∈ [κ δ , max ]. Theorem 4.4 generalizes the known bounds for the cases where F = R and q = 1 [46], q = 2 [16,47] and q = 3 [1]. The results for q = 2 with F ⊂ R n and for q > 3 appear to be new.…”

“…From the complexity point of view, it is known that the complexity of obtaining -approximate first-order criticality for unconstrained and convexly constrained problem can be reduced to O( −( p+1)/ p ) if one is ready to define the step by using a regularization model of order p ≥ 1. In the unconstrained case, this was shown for p = 2 in [16,47] and for general p ≥ 1 in [9], while the convexly constrained case was analysed (for p = 2) in [17]. The question of whether this methodology and the associated improvements in evaluation complexity bounds can be extended to order above one also remains open at this stage.…”

“…(used in [16,47] for instance). Indeed, for one-dimensional problems and assuming ∇ 2 x f (x) ≤ 0, the former condition amounts to requiring that…”

Worst case complexity of direct search under convexity

“…We know from [17] that, in this bound, only the minimum cosine measure of the positive spanning sets depends explicitly on n. One also knows from the positive spanning set formed by the coordinate vectors and their negatives that such minimum cosine measure can be set greater than or equal to 1/ √ n, and thus 1/ω ≤ O(np 2 ), where ω is given in (7). On the other hand, each poll step when using such positive spanning sets costs at most O(n) function evaluations.…”

“…Such a bound has been proved sharp or tight by Cartis, Gould, and Toint [1]. There has been quite an amount of research on WCC bounds for several other classes of algorithms in the non-convex case (see, e.g., [7,9,14]). …”