A second-order globally convergent direct-search method and its worst-case complexity

Abstract

Direct-search algorithms form one of the main classes of algorithms for smooth unconstrained derivative-free optimization, due to their simplicity and their well-established convergence results. They proceed by iteratively looking for improvement along some vectors or directions. In the presence of smoothness, first-order global convergence comes from the ability of the vectors to approximate the steepest descent direction, which can be quantified by a first-order criticality (cosine) measure. The use of a set of vectors with a positive cosine measure together with the imposition of a sufficient decrease condition to accept new iterates leads to a convergence result as well as a worst-case complexity bound. In this paper, we present a second-order study of a general class of direct-search methods. We start by proving a weak second-order convergence result related to a criticality measure defined along the directions used throughout the iterations. Extensions of this result to obtain a true second-order optimality one are discussed, one possibility being a method using approximate Hessian eigenvectors as directions (which is proved to be truly secondorder globally convergent). Numerically guaranteeing such a convergence can be rather expensive to ensure, as it is indicated by the worst-case complexity analysis provided in this paper, but turns out to be appropriate for some pathological examples.

Derivative-free optimization methods

“…Gratton et al. (2016) then demonstrate that this augmentation of Algorithm 2has a subsequence that converges to a second-order stationary point. That is, they prove a convergence result of the form (2.7) and demonstrate a WCC result of type (2.3) in (see Table A.1).…”

“…When f ∈ LC 2 , work by Gratton et al [2016] essentially augments the DDS method analysed by Vicente [2013], but forms an approximate Hessian via central differences from function evaluations obtained (for free) by using a particular choice of D k . Gratton et al [2016] then demonstrate that this augmentation of Algorithm 2 has a subsequence that converges to a second-order stationary point. That is, they prove a convergence result of the form (7) and demonstrate a WCC result of type (3) in O( −3 ) (see Table 8.1).…”

“…When , work by Gratton et al. (2016) essentially augments the DDS method analysed by Vicente (2013), but forms an approximate Hessian via central differences from function evaluations obtained (for free) by using a particular choice of . Gratton et al.…”

“…Section 2 presents deterministic methods for solving (DET) when . The section is split between direct-search methods and model-based methods, although the lines between these are increasingly blurred; see, for example, Conn and Le Digabel (2013), Custódio, Rocha and Vicente (2009), Gramacy and Le Digabel (2015) and Gratton, Royer and Vicente (2016). Direct-search methods are summarized in far greater detail by Kolda, Lewis and Torczon (2003) and Kelley (1999 b ), and in the more recent survey by Audet (2014).…”

“…1 − p1 DDS [Gratton et al, 2015] C mnL 2 g (f (x0) − f (x * )) 2 ∇f (x k ) ≤ w.p. 1 − p2 TR [Gratton et al, 2018] C,D m max{κ ef , κeg} 2 (f (x0) − f (x * )) 2 f ∈ LC 2 max{ ∇f (x k ) , −λ k } ≤ DDS [Gratton et al, 2016] n 5 max{LH, Lg} 3 (f (x0) − f (x * )) 3 TR[Gratton et al, 2019a] n 5 max{L 3 H , L 2 g }(f (x0) − f (x * )) 3 max{ ∇f (x k ) , −λ k } ≤ w.p. 1 − p3 TR [Gratton et al, 2018] C,D m max{κeg, κeH} 3 (f (x0) − f (x * ))…”