Optimal Methods for Higher-Order Smooth Monotone Variational Inequalities

Abstract

In this work, we present new simple and optimal algorithms for solving the variational inequality (VI) problem for p th -order smooth, monotone operators -a problem that generalizes convex optimization and saddle-point problems. Recent works (Bullins and Lai (2020), Lin and Jordan (2021), Jiang and Mokhtari ( 2022)) present methods that achieve a rate of O(ε −2/(p+1) ) for p ≥ 1, extending results by (Nemirovski ( 2004)) and (Monteiro and Svaiter ( 2012)) for p = 1, 2. A drawback to these approaches, however, is their reliance on a line search scheme. We provide the first p th -order method that achieves a rate of O(ε −2/(p+1) ). Our method does not rely on a line search routine, thereby improving upon previous rates by a logarithmic factor. Building on the Mirror Prox method of Nemirovski (2004), our algorithm works even in the constrained, non-Euclidean setting. Furthermore, we prove the optimality of our algorithm by constructing matching lower bounds. These are the first lower bounds for smooth MVIs beyond convex optimization for p > 1. This establishes a separation between solving smooth MVIs and smooth convex optimization, and settles the oracle complexity of solving p th -order smooth MVIs.

Optimal and Adaptive Monteiro-Svaiter Acceleration

“…Finally, we report empirical results (Section 4). 1 On logistic regression problems, combining our optimal acceleration scheme with our adaptive oracle outperforms previously proposed accelerated second-order methods. However, we also show that (while somewhat helpful for O cr with a conservative choice of H), adding momentum to well-tuned or adaptive second-order methods is harmful in logistic regression: simply iterating our oracle-or, better yet, applying Newton's method-dramatically outperforms all "accelerated" algorithms.…”

“…These works also feature an implicit equation over a scalar regularization/step-size parameter, that necessitates a bisection and increases complexity by a logarithmic factor. In recent papers, Lin and Jordan [29] and Adil et al [1] remove that logarithmic factor by developing bisection-free methods for variational inequalities. However, applying these methods directly to convex optimization with Lipschitz pth derivatives yields a rate of O(t −(p+1)/2 ) rather than the optimal O(t −(3p+1)/2 ) rate of our method.…”

“…In the next section, we will show how to build MS oracles that, given query y, output (x, λ) such that (x, y, λ) always satisfy a (s, c)-movement bound, for certain s and c depending on the oracle type and function structure (e.g., level of smoothness). For example, when f has H-Lipschitz Hessian, the cubic-regularized Newton step with M = 2H is a 1 2 -MS oracle that guarantees a (2, √ H)-movement bound. With the necessary definitions in hand, we are ready to state our main result: the iteration (and MS oracle query) complexity of Algorithm 1.…”

“…Nevertheless, Carmon et al [12,Alg. 3] describe a method O r-BaCoN based on solving a sequence of O(1) trust-region problems (ball-constrained Newton steps), which we call that, for functions that are O(1)-Hessian stable in a ball of radius r (or 1/r-quasi-self-concordant) and have a finite condition number, outputs (x, λ) satisfying the 1 2 -MS oracle and an (∞, O(1/r)) movement bound. Implementing O r-BaCoN requires only a single Hessian evaluation and a number of linear system solutions that is polylogarithmic in problem parameters.…”

“…Their algorithm is based on the fact that many oracles, including O cr , implicitly approximate proximal points. That is, for every y and x = O(y), there exists λ x,y > 0 such that x ≈ argmin x ∈X f (x ) + 1 2 λ x,y x − y 2 , with the approximation error controlled by a specific condition they define. MS prove that, under this condition, the accelerated proximal point method [23,39] (with dynamic regularization parameter) maintains its rate of convergence.…”