Robust Accelerated Primal-Dual Methods for Computing Saddle Points

Abstract

We consider strongly convex/strongly concave saddle point problems assuming we have access to unbiased stochastic estimates of the gradients. We propose a stochastic accelerated primal-dual (SAPD) algorithm and show that SAPD iterate sequence, generated using constant primal-dual step sizes, linearly converges to a neighborhood of the unique saddle point, where the size of the neighborhood is determined by the asymptotic variance of the iterates. Interpreting the asymptotic variance as a measure of robustness to gradient noise, we obtain explicit characterizations of robustness in terms of SAPD parameters and problem constants. Based on these characterizations, we develop computationally tractable techniques for optimizing the SAPD parameters, i.e., the primal and dual step sizes, and the momentum parameter, to achieve a desired trade-off between the convergence rate and robustness on the Pareto curve. This allows SAPD to enjoy fast convergence properties while being robust to noise as an accelerated method. We also show that SAPD admits convergence guarantees for the gap metric with a variance term optimal up to a logarithmic factor -which can be removed by employing a restarting strategy. Furthermore, to our knowledge, our work is the first one showing an iteration complexity result for the gap function on smooth SCSC problems without the bounded domain assumption. Finally, we illustrate the efficiency of our approach on distributionally robust logistic regression problems.

Smooth monotone stochastic variational inequalities and saddle point problems: A survey

“…A gap also appears for the SPP with stochastic finite-sum structure [58,82,118]. The stochastic setting with bounded variance was considered in [32,85,125].…”

The Power of First-Order Smooth Optimization for Black-Box Non-Smooth Problems

“…By using the restart technique we can generalize these results to µ-strongly convex, µ-strongly concave case, see Section 3.3. Alternatively, we can combine the Smoothing technique with Stochastic Accelerated Primal-Dual method from [75] for (12) with f (x, y) being µ x -strongly convex and µ y -strongly concave in 2-norm (Euclidean setup). In this case we obtain the following bounds…”

“…Note, that most of the results for saddle-point problems (i.e. mentioned result from [75] or finite-sum composite generalization [69]) with different constants of smoothness and strong convexity/concavity were obtained based on Accelerated gradient method for convex problems and Catalyst envelope, that allows to generalize it to saddle-point problems [47]. There exist also loop-less (direct) accelerated methods that save ln(ε −1 )-factor in the complexity, for µ x -strongly convex, µ y -strongly concave saddle-point problems [40].…”

“…In particular, this allows to obtain new methods for stochastic bilinear saddle-point problems with composites based on the state-of-the-art method of [40]. The latter method, for the considered class of problems, works better than Stochastic Accelerated Primal-Dual method [75] which we use in Section 3.4. Interestingly, our Smoothing scheme is not the only technique for reduction of algorithms for smooth convex problems to algorithms for non-smooth convex problems.…”