A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization

Abstract

In this paper, we analyze several methods for approximating the gradient of a function using only function values. These methods include finite differences, linear interpolation, Gaussian smoothing and smoothing on a unit sphere. The methods differ in the number of functions sampled, the choice of the sample points and the way in which the gradient approximations are derived. For each method, we derive bounds on the number of samples and the sampling radius which guarantee favorable convergence properties for a line search or fixed step size descent method. To this end, we derive one common condition on the accuracy of gradient approximations which guarantees these convergence properties and then show how each method can satisfy this condition. We analyze the convergence properties even when this condition is only satisfied with some sufficiently large probability at each iteration, as happens to be the case with Gaussian smoothing and smoothing on a unit sphere. Finally, we present numerical results evaluating the quality of the gradient approximations as well as their performance in conjunction with a line search derivative-free optimization algorithm.

Scalable Subspace Methods for Derivative-Free Nonlinear Least-Squares Optimization

“…This framework has lead to variants of methods such as stochastic gradient descent [34], SVRG [51] and Adam [22], for example. We note that linear interpolation to orthogonal directions-more similar to traditional model-based DFO-has been shown to outperform gradient sampling as a gradient estimation technique [8,7].…”

“…This gives a total cost of O(p 3 + np). 8 Alternative Point Removal Mechanism Instead of Algorithm 4, we could have used a simpler mechanism for removing points, such as removing the points furthest from the current iterate (with total cost 9 O(np)). However, this leads to a substantial performance penalty.…”

Accelerated gradient methods with absolute and relative noise in the gradient

“…That is far from what we've declared in section 1. To improve this it's worth to propose more detailed conception rather then (7).…”

“…particular, we propose an alternative approach to regularization § that is based on "early stopping" ¶ of considered iterative procedure by developing proper stopping rule. Now we explain how to reduce relative inexactness (3) to (7) and to apply (9) when µ ε. Since f (x) has Lipschitz gradient from (3), (7) we may derive that after k iterations (where k is greater than L/µ on a logarithmic factor log LR 2 /ε , where ε -accuracy in function value)…”

“…Note, that the last relation is a consequence of the previous ones and in general is not equivalent to them [49,25]. In many applications, especially for gradient-free methods (when estimating the gradient by finite differences [11,44,7]) optimization problems in infinite dimensional spaces (such examples arise when solving inverse problems [31,26]) instead of an access to ∇f (x) we have an access to its inexact approximation ∇f (x).…”