A decoupled first/second-order steps technique for nonconvex nonlinear unconstrained optimization with improved complexity bounds

Abstract

In order to be provably convergent towards a second-order stationary point, optimization methods applied to nonconvex problems must necessarily exploit both first and second-order information. However, as revealed by recent complexity analyses of some of these methods, the overall effort to reach second-order points is significantly larger when compared to the one of approaching first-order ones. On the other hand, there are other algorithmic schemes, initially designed with first-order convergence in mind, that do not appear to maintain the same first-order performance when modified to take second-order information into account.In this paper, we propose a technique that separately computes first and second-order steps, and that globally converges to second-order stationary points: it consists in better connecting the steps to be taken and the stationarity criteria, potentially guaranteeing larger steps and decreases in the objective. Our approach is shown to lead to an improvement of the corresponding complexity bound with respect to the first-order optimality tolerance, while having a positive impact on the practical behavior. Although the applicability of our ideas is wider, we focus the presentation on trust-region methods with and without derivatives, and motivate in both cases the interest of our strategy.

A concise second-order complexity analysis for unconstrained optimization using high-order regularized models

“…Specifically, if χ f,1 (x k ) > ǫ 1 , one might simply require that χ m,1 (x k , s k , σ k ) ≤ θ s k p rather than (2.20) as this alone would aim to improve first-order criticality. However, though this decoupling is possible both in practice and in the analysis, it is not as straightforward as in the case of say, trust-region methods [12], as the lower bounds on the step in (3.3) and (3.4) depend on the objective's gradient and Hessian value at the next trial point/iterate, not the current x k . Also, one might modify the ARp algorithm to check the optimality measures (2.18) at every trial point, not just successful ones.…”

Complexity Analysis of Second-Order Line-Search Algorithms for Smooth Nonconvex Optimization

“…The difference can be explained * Version of December 12, 2017.by the restriction enforced by the trust-region constraint on the norm of the steps. Recent work has shown that it is possible to improve the bound for trust-region algorithms using specific definitions of the trust-region radius [13]. The best known iteration bound for a second-order algorithm (that is, an algorithm relying on the use of second-order derivatives and Newton-type steps) is O max −3/2 g , −3 H .…”

Derivative-free optimization methods

“…In more recent work, Garmanjani, Jùdice and Vicente (2016) provide a WCC bound of the form (2.2) for Algorithm 3, recovering essentially the same upper bound on the number of function evaluations required by DDS methods found in Vicente (2013), that is, a WCC bound in (see Table A.1). When , Gratton, Royer and Vicente (2019 a ) demonstrate a second-order WCC bound of the form (2.3) in ; in order to achieve this result, fully quadratic models are required. In Section 3.3, a similar result is achieved by using randomized variants that do not require a fully quadratic model in every iteration.…”

“…1 − p1 DDS [Gratton et al, 2015] C mnL 2 g (f (x0) − f (x * )) 2 ∇f (x k ) ≤ w.p. 1 − p2 TR [Gratton et al, 2018] C,D m max{κ ef , κeg} 2 (f (x0) − f (x * )) 2 f ∈ LC 2 max{ ∇f (x k ) , −λ k } ≤ DDS [Gratton et al, 2016] n 5 max{LH, Lg} 3 (f (x0) − f (x * )) 3 TR[Gratton et al, 2019a] n 5 max{L 3 H , L 2 g }(f (x0) − f (x * )) 3 max{ ∇f (x k ) , −λ k } ≤ w.p. 1 − p3 TR [Gratton et al, 2018] C,D m max{κeg, κeH} 3 (f (x0) − f (x * ))…”