Distributed Saddle-Point Problems: Lower Bounds, Near-Optimal and Robust Algorithms

Beznosikov, Aleksandr; Samokhin, Valentin; Gasnikov, Alexander

doi:10.48550/arxiv.2010.13112

Cited by 18 publications

(58 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where in each iteration, the same sample ξ k is used for the exploration (computation of x k − γ 1,ξ k F ξ k (x k )) and update (computation of x k+1 ) steps. The selection of step-sizes γ 2,ξ k and γ 1,ξ k that guarantee convergence of the method in different settings varies across previous papers [Mishchenko et al, 2020, Beznosikov et al, 2020, Hsieh et al, 2019. In this work, the proposed stepsizes for S-SEG satisfy 0 < γ 2,ξ k = αγ 1,ξ k , where 0 < α < 1, and are allowed to depend on the sample ξ k .…”

Section: Preliminariesmentioning

confidence: 99%

“…2 Typically, S-SEG is analyzed under Lipschitzness and (strong) monotonicity of individual stochastic realizations F ξ [Mishchenko et al, 2020] that are stronger than Assumptions 1.1 and 1.2. In contrast, I-SEG is studied under Assumptions 1.1 and 1.2 but with additional assumptions like uniformly bounded variance or its relaxations [Beznosikov et al, 2020, Hsieh et al, 2020.…”

Section: Preliminariesmentioning

confidence: 99%

“…4.1; b = batchsize. Assumptions on constant stepsizes: Mishchenko et al [2020] uses γ ≤ 1 /2Lmax, Hsieh et al [2019] uses γ 2 ≤ γ 1 ≤ c /L for some positive c > 0, Beznosikov et al [2020] uses γ ≤ 1 /4L, and we use γ…”

Section: Contributionsmentioning

confidence: 99%

“…Numerical factors in our theoretical estimates can be tightened for S-SEG when µ i ≥ 0 and for I-SEG when δ = 0. [Beznosikov et al, 2020 Mishchenko et al [2020] consider a regularized version of (VIP) with µ min -strongly convex regularization, F (x) = E ξ [F ξ (x)] and F ξ (x) being monotone and L ξ -Lipschitz. In this case, one can construct an equivalent problem with convex regularizer,…”

Section: Contributionsmentioning

confidence: 99%

“…This discrepancy between the deterministic and the stochastic case has motivated a whole line of work , Mishchenko et al, 2020, Beznosikov et al, 2020, Hsieh et al, 2019 to understand better the properties of SEG. However, several important questions remain open.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Stochastic Extragradient: General Analysis and Improved Rates

Gorbunov¹,

Berard²,

Gidel³

et al. 2021

Preprint

View full text Add to dashboard Cite

The Stochastic Extragradient (SEG) method is one of the most popular algorithms for solving min-max optimization and variational inequalities problems (VIP) appearing in various machine learning tasks. However, several important questions regarding the convergence properties of SEG are still open, including the sampling of stochastic gradients, mini-batching, convergence guarantees for the monotone finite-sum variational inequalities with possibly non-monotone terms, and others. To address these questions, in this paper, we develop a novel theoretical framework that allows us to analyze several variants of SEG in a unified manner. Besides standard setups, like Same-Sample SEG under Lipschitzness and monotonicity or Independent-Samples SEG under uniformly bounded variance, our approach allows us to analyze variants of SEG that were never explicitly considered in the literature before. Notably, we analyze SEG with arbitrary sampling which includes importance sampling and various mini-batching strategies as special cases. Our rates for the new variants of SEG outperform the current state-of-the-art convergence guarantees and rely on less restrictive assumptions.

show abstract

Section: Preliminariesmentioning

confidence: 99%

Section: Preliminariesmentioning

confidence: 99%

Section: Contributionsmentioning

confidence: 99%

Section: Contributionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Stochastic Extragradient: General Analysis and Improved Rates

Gorbunov¹,

Berard²,

Gidel³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Penalty-Based Method for Decentralized Optimization over Time-Varying Graphs

Rogozin

Gasnikov

2020

Optimization and Applications

View full text Add to dashboard Cite

We consider a distributed stochastic optimization problem that is solved by a decentralized network of agents with only local communication between neighboring agents. The goal of the whole system is to minimize a global objective function given as a sum of local objectives held by each agent. Each local objective is defined as an expectation of a convex smooth random function and the agent is allowed to sample stochastic gradients for this function. For this setting we propose the first accelerated (in the sense of Nesterov's acceleration) method that simultaneously attains optimal up to a logarithmic factor communication and oracle complexity bounds for smooth strongly convex distributed stochastic optimization. We also consider the case when the communication graph is allowed to vary with time and obtain complexity bounds for our algorithm, which are the first upper complexity bounds for this setting in the literature.

show abstract

Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities

Beznosikov

Gasnikov

2022

Optimization and Applications

View full text Add to dashboard Cite

Variational inequalities are a broad and flexible class of problems that includes minimization, saddle point, fixed point problems as special cases. Therefore, variational inequalities are used in a variety of applications ranging from equilibrium search to adversarial learning. Today's realities with the increasing size of data and models demand parallel and distributed computing for real-world machine learning problems, most of which can be represented as variational inequalities. Meanwhile, most distributed approaches has a significant bottleneck -the cost of communications. The three main techniques to reduce both the total number of communication rounds and the cost of one such round are the use of similarity of local functions, compression of transmitted information and local updates. In this paper, we combine all these approaches. Such a triple synergy did not exist before for variational inequalities and saddle problems, nor even for minimization problems. The methods presented in this paper have the best theoretical guarantees of communication complexity and are significantly ahead of other methods for distributed variational inequalities. The theoretical results are confirmed by adversarial learning experiments on synthetic and real datasets.

show abstract

Distributed Saddle-Point Problems: Lower Bounds, Near-Optimal and Robust Algorithms

Cited by 18 publications

References 0 publications

Stochastic Extragradient: General Analysis and Improved Rates

Stochastic Extragradient: General Analysis and Improved Rates

Penalty-Based Method for Decentralized Optimization over Time-Varying Graphs

Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities

Contact Info

Product

Resources

About