Generalized Optimistic Methods for Convex-Concave Saddle Point Problems

Jiang, Ruichen; Mokhtari, Aryan

doi:10.48550/arxiv.2202.09674

Cited by 3 publications

(9 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our algorithm requires access to an oracle for solving an MVI subproblem (see Definition 3.1) obtained by regularizing the p th -order Taylor series expansion for the operator. This is analogous to the Taylor series oracle from the works on highly-smooth convex optimization , Gasnikov et al, 2019, and identical to the oracle from the Jiang and Mokhtari [2022] work on highly-smooth VIs.…”

Section: Introductionmentioning

confidence: 86%

“…We note that for the case of X = R n , d (x ) = x 2 2 and F = ∇f , where f is a p th -order smooth convex function, the above subproblem is equivalent to the subproblem solved by the algorithm of (up to constant factors), which is known to have optimal iteration complexity for highly-smooth convex optimization. Previous works on higher-order smooth MVIs also solve essentially the same subproblem in their algorithms [Jiang and Mokhtari, 2022]. It has been shown by [Jiang and Mokhtari, 2022, Lemma 7.1] that these subproblems are monotone and are guaranteed to have a unique solution, though efficiently finding such a solution in general remains an open problem, even in the case of convex optimization.…”

Section: Algorithmmentioning

confidence: 99%

“…Under the same second-order smoothness assumption, Monteiro and Svaiter [2012] show how to achieve an improved convergence rate of O(ε −2/3 ). For p th -order smooth MVIs, recent works [Bullins and Lai, 2020, Lin and Jordan, 2021, Jiang and Mokhtari, 2022] have established convergence rates of O(ε −2/(p+1) ), again assuming access to a p th -order oracle. Note that this rate is strictly worse than that for convex optimization.…”

Section: Introductionmentioning

confidence: 99%

“…Approximately Solving Subproblems. The p th -order MVI subproblems (Definition 3.1) that need to be solved in our algorithm are identical to those arising the in the algorithm from Jiang and Mokhtari [2022], and when restricted to the case of unconstrained convex optimization with smoothness measured in Euclidean norms, they become identical to those from . In the appendix, we show that it is sufficient to solve the subproblems approximately.…”

Section: Introductionmentioning

confidence: 99%

“…All previous works on higher-order algorithms [Gasnikov et al, 2019, Jiang and Mokhtari, 2022 assume access to an oracle for solving such subproblems. Even for the special case of unconstrained convex optimization and Euclidean norms, it remains an open problem for how to solve these subproblems for p ≥ 3.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Optimal Methods for Higher-Order Smooth Monotone Variational Inequalities

Adil¹,

Bullins²,

Jambulapati³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this work, we present new simple and optimal algorithms for solving the variational inequality (VI) problem for p th -order smooth, monotone operators -a problem that generalizes convex optimization and saddle-point problems. Recent works (Bullins and Lai (2020), Lin and Jordan (2021), Jiang and Mokhtari ( 2022)) present methods that achieve a rate of O(ε −2/(p+1) ) for p ≥ 1, extending results by (Nemirovski ( 2004)) and (Monteiro and Svaiter ( 2012)) for p = 1, 2. A drawback to these approaches, however, is their reliance on a line search scheme. We provide the first p th -order method that achieves a rate of O(ε −2/(p+1) ). Our method does not rely on a line search routine, thereby improving upon previous rates by a logarithmic factor. Building on the Mirror Prox method of Nemirovski (2004), our algorithm works even in the constrained, non-Euclidean setting. Furthermore, we prove the optimality of our algorithm by constructing matching lower bounds. These are the first lower bounds for smooth MVIs beyond convex optimization for p > 1. This establishes a separation between solving smooth MVIs and smooth convex optimization, and settles the oracle complexity of solving p th -order smooth MVIs.

show abstract

Section: Introductionmentioning

confidence: 86%

Section: Algorithmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Optimal Methods for Higher-Order Smooth Monotone Variational Inequalities

Adil¹,

Bullins²,

Jambulapati³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Optimal and Adaptive Monteiro-Svaiter Acceleration

Carmon¹,

Hausler²,

Jambulapati³

et al. 2022

Preprint

View full text Add to dashboard Cite

We develop a variant of the Monteiro-Svaiter (MS) acceleration framework that removes the need to solve an expensive implicit equation at every iteration. Consequently, for any p ≥ 2 we improve the complexity of convex optimization with Lipschitz pth derivative by a logarithmic factor, matching a lower bound. We also introduce an MS subproblem solver that requires no knowledge of problem parameters, and implement it as either a second-or first-order method via exact linear system solution or MinRes, respectively. On logistic regression our method outperforms previous second-order acceleration schemes, but under-performs Newton's method; simply iterating our first-order adaptive subproblem solver performs comparably to L-BFGS.∞ regression [8,12], minimizing functions with Hölder continuous higher derivatives [40], and distributionally-robust optimization [13,11].

show abstract

Convergence rate analysis of the gradient descent-ascent method for convex-concave saddle-point problems

Zamani¹,

Abbaszadehpeivasti²,

Klerk³

2022

Preprint

View full text Add to dashboard Cite

In this paper, we study the gradient descent-ascent method for convex-concave saddle-point problems. We derive a new non-asymptotic global convergence rate in terms of distance to the solution set by using the semidefinite programming performance estimation method. The given convergence rate incorporates most parameters of the problem and it is exact for a large class of strongly convex-strongly concave saddle-point problems for one iteration. We also investigate the algorithm without strong convexity and we provide some necessary and sufficient conditions under which the gradient descent-ascent enjoys linear convergence.

show abstract

Generalized Optimistic Methods for Convex-Concave Saddle Point Problems

Cited by 3 publications

References 27 publications

Optimal Methods for Higher-Order Smooth Monotone Variational Inequalities

Optimal Methods for Higher-Order Smooth Monotone Variational Inequalities

Optimal and Adaptive Monteiro-Svaiter Acceleration

Convergence rate analysis of the gradient descent-ascent method for convex-concave saddle-point problems

Contact Info

Product

Resources

About