Stochastic Zeroth Order Gradient and Hessian Estimators: Variance Reduction and Refined Bias Bounds

Abstract

We study stochastic zeroth order gradient and Hessian estimators for real-valued functions in R n . We show that, via taking finite difference along random orthogonal directions, the variance of the stochastic finite difference estimators can be significantly reduced. In particular, we design estimators for smooth functions such that, if one uses Θ (k) random directions sampled from the Stiefel's manifold St(n, k) and finite-difference granularity δ, the variance of the gradient estimator is bounded by, and the variance of the Hessian estimator is bounded by O. When k = n, the variances become negligibly small. In addition, we provide improved bias bounds for the estimators. The bias of both gradient and Hessian estimators for smooth function f is of order O δ 2 Γ , where δ is the finite-difference granularity, and Γ depends on high order derivatives of f . Our results are evidenced by empirical observations.

Convergence Rates of Stochastic Zeroth-order Gradient Descent for Łojasiewicz Functions

“…This section is devoted to proving Theorem 3. The proof mimics that for Theorem 2 [11]. To start with, we need the following facts in Propositions 7 and 8.…”

“…Let C 0 and T 0 be two constants so that Proposition 3 is true. By (11) and the Łojasiewicz inequality in Remark 2, with probability 1 it holds that, for all t ≥ T 0 ,…”

“…Let C 0 and T 0 be two constants so that Proposition 3 holds true. Taking total expectation on both sides of (11) gives, for sufficiently large t,…”

“…is uniformly sampled from the Stiefel's manifold St(n, k) := {V ∈ R n×k : V V = I k }, and δ t := 2 −t is the finite difference granularity. Previously, the statistical properties of (2) have been investigated by [11] (See Section 3) and [16] have consider stochastic zeroth order optimization algorithms for Polyak-Łojasiewicz functions (See Section 2 for detailed discussions). Throughout, we use Stochastic Zeroth-order Gradient Descent (SZGD) to refer to the above update rule (1) with the estimator (2).…”