Lower bounds for finding stationary points II: first-order methods

Abstract

We establish lower bounds on the complexity of finding -stationary points of smooth, nonconvex high-dimensional functions using first-order methods. We prove that deterministic firstorder methods, even applied to arbitrarily smooth functions, cannot achieve convergence rates in better than −8/5 , which is within −1/15 log 1 of the best known rate for such methods. Moreover, for functions with Lipschitz first and second derivatives, we prove no deterministic first-order method can achieve convergence rates better than −12/7 , while −2 is a lower bound for functions with only Lipschitz gradient. For convex functions with Lipschitz gradient, accelerated gradient descent achieves the rate −1 log 1 , showing that finding stationary points is easier given convexity.

Lower bounds for finding stationary points I

“…In this section we give a lower bound on the complexity of this function class that has the same dependence as our bound for the class F p (∆, L p ). This is in sharp contrast to convex optimization, where distance-bounded functions enjoy significantly better dependence than their value-bounded counterparts (see Section 3 in the companion [14]). Qualitatively, the reason for this difference is that the lack of convexity allows us to "hide" global minima close to the origin that are difficult to find for any algorithm with local function access [35].…”

“…[10,35,37]). In Part II of this paper [14], we consider the impact of convexity on the difficulty of finding stationary points using first-order methods.…”

“…These results say little about the potential of first-order methods on functions with higher-order Lipschitz derivatives, where first-order methods attain rates better than −2 [13]. In Part II of this series [14], we address this issue and show lower bounds for deterministic algorithms using only firstorder information. The lower bounds exhibit a fundamental gap between first-and second-order methods, and nearly match the known upper bounds [13].…”

“…In Section 2 we introduce the classes of functions and algorithms we consider as well as our notion of complexity. Then, in Section 3, we present the generic technique we use to prove lower bound for deterministic algorithms in both this paper and Part II [14]. While essentially present in previous work, our technique abstracts away and generalizes the central arguments in many lower bounds [35,34,45,4].…”

“…In this paper and its companion [14], we focus on the converse problem: providing dimensionfree complexity lower bounds for finding -stationary points. We show fundamental limits on the best achievable dependence, as well as dependence on other problem parameters.…”