Zeroth-Order Regularized Optimization (ZORO): Approximately Sparse Gradients and Adaptive Sampling

Abstract

We consider the problem of minimizing a high-dimensional objective function, which may include a regularization term, using (possibly noisy) evaluations of the function. Such optimization is also called derivative-free, zeroth-order, or black-box optimization. We propose a new Zeroth-Order Regularized Optimization method, dubbed ZORO. When the underlying gradient is approximately sparse at an iterate, ZORO needs very few objective function evaluations to obtain a new iterate that decreases the objective function. We achieve this with an adaptive, randomized gradient estimator, followed by an inexact proximal-gradient scheme. Under a novel approximately sparse gradient assumption and various different convex settings, we show the (theoretical and empirical) convergence rate of ZORO is only logarithmically dependent on the problem dimension. Numerical experiments show that ZORO outperforms the existing methods with similar assumptions, on both synthetic and real datasets.

Curvature-Aware Derivative-Free Optimization

“…Similar ideas can be found in [26,27,28]. More recently, [29] [30,31,6,32,33,34] for recent progress in overcoming this.…”

“…This setting is known as derivative-free optimization (DFO). DFO has a long tradition and recently revived for reinforcement learning [1,2,3], hyperparameter tuning [4], and adversarial attacks on neural network based classifiers [5,6]. Common to all such applications is the assumption that evaluating f (x) is expensive, time consuming or inconvenient.…”

Communication-Efficient Distributed SGD with Compressed Sensing

“…Besides the related work discussed above, it is worth noting that a recent paper [19] uses compressed sensing for zeroth-order optimization, which exhibits a mathematical structure similar to this study. However, [19] considers the centralized setting and only establishes convergence to a neighborhood of the minimizer.…”

A Dimension-Insensitive Algorithm for Stochastic Zeroth-Order Optimization

“…Such undesirable characteristics render many efficient algorithms, such as the stochastic first-order methods, no longer directly applicable. As a remedy to this, zeroth-order optimization (ZOO), also known as black-box or derivative-free optimization [4], has attracted the attention of many researchers.…”

“…Closely related to this work is the gradient estimation-based ZOO framework discussed by seminal works such as [14,18,1,6,7], where theoretical underpinnings for both convex and nonconvex cases have been established. However, as pointed out by [4,3], the performance of almost all existing ZOO algorithms deteriorates rapidly as the problem dimensionality d increases. In particular, for convex ZOO with fixed σ and L, such an algorithm achieves a query complexity O(d 2 /ǫ 2 ) of the zeroth-order oracles 1 according to [14].…”