Optimal Primal-Dual Methods for a Class of Saddle Point Problems

Abstract

We present a novel accelerated primal-dual (APD) method for solving a class of deterministic and stochastic saddle point problems (SPP). The basic idea of this algorithm is to incorporate a multi-step acceleration scheme into the primaldual method without smoothing the objective function. For deterministic SPP, the APD method achieves the same optimal rate of convergence as Nesterov's smoothing technique. Our stochastic APD method exhibits an optimal rate of convergence for stochastic SPP not only in terms of its dependence on the number of the iteration, but also on a variety of problem parameters. To the best of our knowledge, this is the first time that such an optimal algorithm has been developed for stochastic SPP in the literature. Furthermore, for both deterministic and stochastic SPP, the developed APD algorithms can deal with the situation when the feasible region is unbounded, as long as a saddle point exists. In the unbounded case, we incorporate the modified termination criterion introduced by Monteiro and Svaiter in solving SPP problem posed as monotone inclusion, and demonstrate that the rate of convergence of the APD method depends on the distance from the initial point to the set of optimal solutions.

An Accelerated Linearized Alternating Direction Method of Multipliers

“…. , N − 1 do We may also linearize Bw t − Kx − b 2 , and generate x t+1 by 13) as discussed in [18,10]. This variant is called the preconditioned ADMM (P-ADMM).…”

“…It should be noted that all the methods in the above list require more assumptions on the AECCO and UCO problems (e.g., simplicity of G(·), strong convexity of G(·) or F (·)), in comparison with Nesterov's smoothing scheme. More recently, we proposed an accelerated primal-dual (APD) method for solving the UCO problem [13], which has the same optimal rate of convergence (1.16) as that of Nesterov's smoothing scheme in [40]. The advantage of the APD method over Nesterov's smoothing scheme is that it does not require boundedness on either X or Y .…”

“…Secondly, ALP-ADMM has the same rate of convergence as Nesterov's smoothing scheme [40], and achieves optimal rate of convergence (1.16). Moreover, we can see from (2.36) that the APD method in [13] is equivalent to ALP-ADMM. Nonetheless, AL-ADMM has better constant in the estimation of rate of convergence than both ALP-ADMM and Nesterov's smoothing scheme, since D X,K ≤ K D X .…”

Distributed and Consensus Optimization for Non-smooth Image Reconstruction

“…The linearization idea can also be adopted in the primal-dual framework to overcome the non-simplicity of F(x) term (1.1). In particular, Chen et al [12] developed an accelerated scheme using the Nesterov's idea [34][35][36] …”

On the global convergence rate of the gradient descent method for functions with Hölder continuous gradients

“…These assumptions are usually presented in the form of Lipschitz continuity. Traditionally, convergence properties of most first order methods [4,24,30] including gradient methods [19,31,32] are derived by imposing some Lipschitz continuity assumptions. For a smooth function, it is assumed on the magnitude of its gradient, while for a nonsmooth objective function it is imposed on its values.…”