Small Errors in Random Zeroth Order Optimization are Imaginary

Abstract

The vast majority of zeroth order optimization methods try to imitate first order methods via some smooth approximation of the gradient. Here, the smaller the smoothing parameter, the smaller the gradient approximation error. We show that for the majority of zeroth order methods this smoothing parameter can however not be chosen arbitrarily small as numerical cancellation errors will dominate. As such, theoretical and numerical performance could differ significantly. Using classical tools from numerical differentiation we will propose a new smoothed approximation of the gradient that can be integrated into general zeroth order algorithmic frameworks. Since the proposed smoothed approximation does not suffer from cancellation error, the smoothing parameter (and hence the approximation error) can be made arbitrarily small. Sublinear convergence rates for algorithms based on our smoothed approximation are proved. Numerical experiments are also presented to demonstrate the superiority of algorithms based on the proposed approximation.

Imaginary Zeroth-Order Optimization

“…Building upon the work by [KW52; LM67; NY83; ST98; FKM04; NS17], Jongeneel, Yue, and Kuhn [JYK21] show that randomized zeroth-order optimization also benefits from passing to the complex domain as one can derive an inherently numerically stable method, which is in sharp contrast to common finite-difference methods. This work departs from [JYK21] by introducing an indispensable layer of realism; noise.…”

“…Optimizers, which based on the context could be globally or locally optimal, are denoted by x . We extend [JYK21] and assume that the objective function f can only be accessed through a zeroth-order oracle that outputs corrupted function evaluations at prescribed test points, that is, with noise. As we only have access to such a zeroth-order oracle, our work belongs to the field of zeroth-order optimization, derivative-free optimization or more generally black-box optimization [CSV09;AH17b].…”

“…However, by means of the results in [Pol86] it does appear indirectly in for example the context of reinforcement-learning [Faz+18;Mal+19]. As in [JYK21], Assumption 1.1 is again mainly there to allow for the next assumption. As will be explained below, having access to (f (z)) for some z ∈ C n is at the core of the approach.…”

“…As will be explained below, having access to (f (z)) for some z ∈ C n is at the core of the approach. In contrast to [JYK21] we allow for the presence of (computational) noise.…”

“…In [Duc+15] the authors show the information-theoretic optimality of multi-point (two-point) methods, yet, in [JYK21] the authors show the numerical superiority of single-point schemes. This work sets out to show that this observation prevails when noise is present.…”