Extragradient and Extrapolation Methods with Generalized Bregman Distances for Saddle Point Problems

Abstract

In this work, we introduce two algorithmic frameworks, named Bregman extragradient method and Bregman extrapolation method, for solving saddle point problems. The proposed frameworks not only include the well-known extragradient and optimistic gradient methods as special cases, but also generate new variants such as sparse extragradient and extrapolation methods. With the help of the recent concept of relative Lipschitzness and some Bregman distance related tools, we are able to show certain upper bounds in terms of Bregman distances for "regret" measures. Further, we use those bounds to deduce the convergence rate of O(1/k) for the Bregman extragradient and Bregman extrapolation methods applied to solving smooth convex-concave saddle point problems. Our theory recovers the main discovery made in [Mokhtari et al. (2020), SIAM J. Optim., 20, pp. 3230-3251] for more general algorithmic frameworks with weaker assumptions via a conceptually different approach.

Mirror frameworks for relatively Lipschitz and monotone-like variational inequalities

“…When the mirror mapping function is specialized to the so-called X -regularizer, the mirror EG method recovers the unified mirror prox method in [19]. Formally, the mirror EG method is very similar to the original EG method, which helps us clearly distinguish Nemirovski's mirror prox method [33], Nesterov's dual extrapolation method [35], and our Bregman EG method [43]. The mirror EP method is formally similar to Bregman EP method and covers the operator EP method [23].…”

“…The proposed relative Lipschitzness is well-suited for the standard analyses of extragradient-type methods, as illustrated in [11] for mirror prox and dual extrapolation methods and in [43] for Bregman extragradient and extrapolation methods. In this study, in order to apply the relative Lipschitzness more widely, we slightly modify it by allowing the distance generating function ω to be nondifferentiable.…”

“…The extension is not trivial since the formulation in VIs is quiet different from that in optimization. It has been proved that the relative Lipschitzness is a very suitable tool to convergence analysis for several extragradient-type methods applied to monotone VIs [11,43].…”

“…The new frameworks mainly depend on a mirror mapping function ω. By exploiting the subdifferentiable of ω, we greatly expand our previously proposed Bregman EG and EP methods [43]. When the mirror mapping function is specialized to the so-called X -regularizer, the mirror EG method recovers the unified mirror prox method in [19].…”

Stochastic Variance Reduction for Variational Inequality Methods

“…FoRB only needs one proximal operator, applied to VI. Compared to FBF, FoRB has a simpler update rule and, unlike FBF, it is easy to adjust to Bregman setting, see [AMC20;Zha21].…”