Higher-order methods for convex-concave min-max optimization and monotone variational inequalities

Abstract

We provide improved convergence rates for constrained convex-concave min-max problems and monotone variational inequalities with higher-order smoothness. In min-max settings where the p th -order derivatives are Lipschitz continuous, we give an algorithm HigherOrderMirrorProx that achieves an iteration complexity of O(1/T p+1 2 ) when given access to an oracle for finding a fixed point of a p th -order equation. We give analogous rates for the weak monotone variational inequality problem. For p > 2, our results improve upon the iteration complexity of the first-order Mirror Prox method of Nemirovski [2004] and the second-order method of Monteiro and Svaiter [2012]. We further instantiate our entire algorithm in the unconstrained p = 2 case.

Generalized Optimistic Methods for Convex-Concave Saddle Point Problems

“…Comparison with [MS12] and [BL20]. Similar complexity bounds are also reported in [MS12; BL20] for extragradient-type second-order methods.…”

“…Comparison with [BL20]. We note that similar iteration complexity bounds are also reported in [BL20] for extragradient-type higher-order methods. However, the authors in [BL20] did not discuss how to solve the subproblem (which can be non-monotone) or how to select a stepsize that satisfies their specified conditions.…”

“…Later, it was further extended to the composite objective with the same convergence rate [MS12]. More recently, the authors in [BL20] proposed second-order and higher-order methods by directly extending the analysis of the classical mirrorprox method [Nem04]. They proved that the proposed p-th-order method enjoys a complexity bound of O(ǫ − 2 p+1 ) for smooth convex-concave saddle point problems with Lipschitz p-th-order…”

“…In comparison, our method is based on a different algorithmic idea and has a lower per-iteration computational cost, since it only requires the gradient information at one point and solves one single subproblem instead of two in each iteration. Moreover, in [MS12] the authors only considered the Euclidean setup, while [BL20] only discussed implementation details for the special case of unconstrained problems under stronger assumptions (F is L 1 -Lipschitz and strongly monotone).…”

“…We close the discussion on our iteration complexity by mentioning that similar complexity bounds to (65) and (66) were also established in [OKDG20] but via a very different approach. Specifically, the authors of that paper proposed to apply a restarting technique on the extragradient-type method in [BL20]. Under Assumptions 2.2 and 6.1, they proved a complexity bound of O(( L 2 R µ )…”