Convergence properties of an Objective-Function-Free Optimization regularization algorithm, including an $\mathcal{O}(ε^{-3/2})$ complexity bound

Abstract

An adaptive regularization algorithm for unconstrained nonconvex optimization is presented in which the objective function is never evaluated, but only derivatives are used. This algorithm belongs to the class of adaptive regularization methods, for which optimal worst-case complexity results are known for the standard framework where the objective function is evaluated. It is shown in this paper that these excellent complexity bounds are also valid for the new algorithm, despite the fact that significantly less information is used. In particular, it is shown that, if derivatives of degree one to p are used, the algorithm will find a ǫ 1 -approximate first-order minimizer in at most O(ǫ −(p+1)/p 1 ) iterations, and an (ǫ 1 , ǫ 2 )-approximate second-order minimizer in at most O(max[ǫ −(p+1)/p , ǫ −(p+1)/(p−1) 2 ]) iterations. As a special case, the new algorithm using first and second derivatives, when applied to functions with Lipschitz continuous Hessian, will find an iterate x k at which the gradient's norm is less than ǫ 1 in at most O(ǫ −3/2 1 ) iterations.

Yet another fast variant of Newton's method for nonconvex optimization

“…Promising lines of for future work include inexact derivatives, estimating the regularization parameter without evaluating the objective function (as in [24]), stochastic variants and the handling of simple constraints such as bounds on the variables in the spirit of [12,Section 14.2].…”

“…For nonconvex optimization, these latter variants exhibit a worst-case O −3/2 complexity order to find an -first-order minimizer compared with the O −2 order of second-order trust-region methods [26], [12,Section 3.2]. Adaptive cubic regularization was later extended to handle inexact derivatives [40,41,2,1], probabilistic models [1,13], and even schemes in which the value of the objective function is never computed [24]. However, as noted in [33], the improvement in complexity has been obtained by trading the simple Newton step requiring only the solution of a single linear system for more complex or slower procedures, such as secular iterations, possibly using Lanczos preprocessing [6,8] (see also [12,Chapters 8 to 10]) or (conjugate-)gradient descent [29,4].…”

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

“…The second is that simplicity is achieved by avoiding the computation of the objective-function values and, most commonly, of other derivatives than gradients (hence their name). This in turn has made them very robust in the presence of noise on the function and its derivatives [22], an important feature when the problem is so large that these quantities can only be realistically estimated (typically by sampling) rather than calculated exactly. The context in which optimization is performed with computing function values is sometimes denoted by OFFO (Objective-Function-Free Optimization).…”