Curvature-Aware Derivative-Free Optimization

Abstract

We propose a new line-search method, coined Curvature-Aware Random Search (CARS), for derivative-free optimization. CARS exploits approximate curvature information to estimate the optimal step-size given a search direction. We prove that for strongly convex objective functions, CARS converges linearly if the search direction is drawn from a distribution satisfying very mild conditions. We also explore a variant, CARS-NQ, which uses Numerical Quadrature instead of a Monte Carlo method when approximating curvature along the search direction. We show CARS-NQ is effective on highly non-convex problems of the form f = f cvx + f osc where f cvx is strongly convex and f osc is rapidly oscillating. Experimental results show that CARS and CARS-NQ match or exceed the state-of-the-arts on benchmark problem sets.

Adapting Zeroth Order Algorithms for Comparison-Based Optimization

“…To gain a clearer perspective on how these CBO algorithms compare over the entire benchmark set, we use performance profiles [7]. As described in [16], performance profiles are constructed as follows. Let P denote the set of benchmark problems and S denote the set of algorithms under consideration.…”