On lower complexity bounds for large-scale smooth convex optimization

Abstract

We derive lower bounds on the black-box oracle complexity of large-scale smooth convex minimization problems, with emphasis on minimizing smooth (with Hölder continuous, with a given exponent and constant, gradient) convex functions over high-dimensional · p -balls, 1 ≤ p ≤ ∞. Our bounds turn out to be tight (up to logarithmic in the design dimension factors), and can be viewed as a substantial extension of the existing lower complexity bounds for large-scale convex minimization covering the nonsmooth case and the "Euclidean" smooth case (minimization of convex functions with Lipschitz continuous gradients over Euclidean balls). As a byproduct of our results, we demonstrate that the classical Conditional Gradient algorithm is near-optimal, in the sense of Information-Based Complexity Theory, when minimizing smooth convex functions over high-dimensional · ∞-balls and their matrix analogies -spectral norm balls in the spaces of square matrices.

Optimal Affine-Invariant Smooth Minimization Algorithms

“…When Q is an p ball, with 1 ≤ p ≤ 2, we show that this complexity bound matches lower complexity bounds from Guzmán and Nemirovski (2015) for the large-scale regime (more precisely, n = Ω(1/ε 2 )).…”

“…Before doing this, it is worth mentioning that this optimality only holds for large-scale problems, namely where dimension n is larger than the number of iterations T : if one can afford a superlinear (in dimension) number of iterations, methods such as the center of gravity or ellipsoid can achieve better complexity estimates (Nemirovskiǐ and Yudin, 1979). It was proved in Guzmán and Nemirovski (2015) that the class of problems (16), where f is convex and has L p -Lipschitz continuous gradient w.r.t. · p , satisfies the following lower bound on minimax risk:…”

“…These additional degrees of freedom allow us to match optimal complexity lower bounds from Guzmán and Nemirovski (2015) when optimizing on p balls, with 2 < p < ∞, in the large-scale regime (namely, n = Ω(1/ε 1/[1+2/p] )), 2 with adaptivity in the Hölder smoothness parameter and Lipschitz constant as a bonus. This means that, on p -balls at least, our complexity bounds are optimal not only in terms of target precision ε, but also in terms of smoothness and problem dimension.…”

“…We show that the proposed methods are nearly optimal in the large-scale regime. For the range 2 < p ≤ ∞, Guzmán and Nemirovski (2015) proved that the class of problems (16), where f is convex and has L p -Lipschitz continuous gradient w.r.t. · p , satisfies the following lower bound on accuracy when the number of steps T ≤ n: when the accuracy ε ≥ Ω(L p /[min{p, log n}n 1+2/p ]).…”

The exact information-based complexity of smooth convex minimization

“…On the other hand, establishing lower bounds on the minimax risk requires a more involved analysis, as the bound needs to hold for any first-order method. Several approaches appear in the literature for establishing lower-bounds, including resisting oracles [9], construction of a "worst-case" function [5,10], and reduction to statistical problems [1,11,13].…”