Convergence Rate Analysis of Primal-Dual Splitting Schemes

Davis, Damek

doi:10.1137/151003076

Cited by 53 publications

(56 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, the convergence rate analysis in section 3 cannot be subsumed by the recent convergence rate analysis of the primal-dual gap of Vũ and Condat's algorithm [15], which only applies when γτ < 1. The original FDRS paper did not show this connection [7, Remark 6.3(iii)].…”

Section: Primal-dual Splittingsmentioning

confidence: 99%

See 1 more Smart Citation

Convergence Rate Analysis of the Forward-Douglas-Rachford Splitting Scheme

Davis¹

2015

SIAM J. Optim.

Self Cite

View full text Add to dashboard Cite

Operator splitting schemes are a class of powerful algorithms that solve complicated monotone inclusion and convex optimization problems that are built from many simpler pieces. They give rise to algorithms in which all simple pieces of the decomposition are processed individually. This leads to easily implementable and highly parallelizable or distributed algorithms, which often obtain nearly state-of-the-art performance. In this paper, we analyze the convergence rate of the forwardDouglas-Rachford splitting (FDRS) algorithm, which is a generalization of the forward-backward splitting (FBS) and Douglas-Rachford splitting algorithms. Under general convexity assumptions, we derive the ergodic and nonergodic convergence rates of the FDRS algorithm, and show that these rates are the best possible. Under Lipschitz differentiability assumptions, we show that the best iterate of FDRS converges as quickly as the last iterate of the FBS algorithm. Under strong convexity assumptions, we derive convergence rates for a sequence that strongly converges to a minimizer. Under strong convexity and Lipschitz differentiability assumptions, we show that FDRS converges linearly. We also provide examples where the objective is strongly convex, yet FDRS converges arbitrarily slowly. Finally, we relate the FDRS algorithm to a primal-dual FBS scheme and clarify its place among existing splitting methods. Our results show that the FDRS algorithm automatically adapts to the regularity of the objective functions and achieves rates that improve upon the sharp worst case rates that hold in the absence of smoothness and strong convexity. 1761The forward-backward splitting (FBS) algorithm [23] is another technique for solving (1.1) when g is known to be smooth. In this case, the proximal operator of g is never evaluated. Instead, FBS combines gradient (forward) steps with respect to g and proximal (backward) steps with respect to f . FBS is especially useful when the proximal operator of g is complex and its gradient is simple to compute.Recently, the forward-Douglas-Rachford splitting (FDRS) algorithm [7] was proposed to combine DRS and FBS and extend their applicability (see Algorithm 1). More specifically, let V ⊆ H be a closed vector space and suppose g is smooth. Then FDRS applies to the following constrained problem:Problem (1.5) arises in the dual form soft-margin kernelized support vector machine classifier [14] in which C is a box constraint, b is 0, and A has rank one. Note that by the argument in (1.3), we can always assume that b = 0. Define the smooth function g(x) := (1/2) Qx, x + c, x , the indicator function f (x) := χ C (x) (which is 0 on C and ∞ elsewhere), and the vector space V := {x ∈ R d | Ax = 0}. With this notation, (1.5) is in the form (1.2) and, thus, FDRS can be applied. This splitting is nice because ∇g(x) = Qx + c is simple whereas the proximal operator of g requires a matrix inversion prox γg = (I R d +γQ) −1 •(I R d −γc), which is expensive for large-scale problems. Goals, challenges, and approaches.This work se...

show abstract

Section: Primal-dual Splittingsmentioning

confidence: 99%

“…Furthermore, it is unclear how the FDRS algorithm relates to other algorithms. We seek to fill this gap.The techniques used in this paper are based on [15,16,17]. These techniques are quite different from those used in classical objective error convergence rate analysis.…”

mentioning

confidence: 99%

Convergence Rate Analysis of the Forward-Douglas-Rachford Splitting Scheme

Davis¹

2015

SIAM J. Optim.

Self Cite

View full text Add to dashboard Cite

show abstract

“…There has been a large body of work analyzing the convergence of ADMM/DRS and PDHG. The analysis of Section 3 utilize proof techniques developed in such past work [7,18,14,49,19].…”

Section: The Main Methodmentioning

confidence: 99%

Splitting with Near-Circulant Linear Systems: Applications to Total Variation CT and PET

Ryu¹,

Ko²,

Won³

2020

SIAM J. Sci. Comput.

View full text Add to dashboard Cite

Many imaging problems, such as total variation reconstruction of X-ray computed tomography (CT) and positron-emission tomography (PET), are solved via a convex optimization problem with near-circulant, but not actually circulant, linear systems. The popular methods to solve these problems, alternating directions method of multipliers (ADMM) and primal-dual hybrid gradient (PDHG), do not directly utilize this structure. Consequently, ADMM requires a costly matrix inversion as a subroutine, and PDHG takes too many iterations to converge. In this paper, we present Near-Circulant Splitting (NCS), a novel splitting method that leverages the near-circulant structure. We show that NCS can converge with an iteration count close to that of ADMM, while paying a computational cost per iteration close to that of PDHG. Through experiments on a CUDA GPU, we empirically validate the theory and demonstrate that NCS can effectively utilize the parallel computing capabilities of CUDA.

show abstract

“…In terms of structure assumption, our first algorithm achieves the same O 1 krate as in [11,16,15,17,25,44,41] without any assumption except for the existence of a saddle point. Moreover, the rate of convergence is on the last iterate, which is important for sparse and low-rank optimization (since averaging essentially destroys the sparsity or low-rankness of the approximate solutions).…”

mentioning

confidence: 97%

“…Existing state-of-the-art primal-dual methods often achieve the best-known O 1 k -rate without strong convexity and Lipschitz gradient continuity, where k is the iteration counter. However, such a rate is often obtained via an ergodic sense or a weighted averaging sequence [11,16,15,17,25,44,41]. Under a stronger condition such as either strong convexity or Lipschitz gradient continuity, one can achieve the best-known O 1 k 2 -convergence rate as shown in, e.g., [11,16,15,17,41].…”

mentioning

confidence: 99%

Proximal alternating penalty algorithms for nonsmooth constrained convex optimization

Tran-Dinh

2018

Comput Optim Appl

View full text Add to dashboard Cite

We develop two new proximal alternating penalty algorithms to solve a wide range class of constrained convex optimization problems. Our approach mainly relies on a novel combination of the classical quadratic penalty, alternating minimization, Nesterov's acceleration, and adaptive strategy for parameters. The first algorithm is designed to solve generic and possibly nonsmooth constrained convex problems without requiring any Lipschitz gradient continuity or strong convexity, while achieving the best-known O 1 k -convergence rate in a non-ergodic sense, where k is the iteration counter. The second algorithm is also designed to solve non-strongly convex, but semi-strongly convex problems. This algorithm can achieve the best-known O 1 k 2convergence rate on the primal constrained problem. Such a rate is obtained in two cases: (i) averaging only on the iterate sequence of the strongly convex term, or (ii) using two proximal operators of this term without averaging. In both algorithms, we allow one to linearize the second subproblem to use the proximal operator of the corresponding objective term. Then, we customize our methods to solve different convex problems, and lead to new variants. As a byproduct, these algorithms preserve the same convergence guarantees as in our main algorithms. We verify our theoretical development via different numerical examples and compare our methods with some existing state-of-the-art algorithms.Keywords Proximal alternating algorithm · quadratic penalty method · accelerated scheme · constrained convex optimization · first-order methods · convergence rate.

show abstract

Convergence Rate Analysis of Primal-Dual Splitting Schemes

Cited by 53 publications

References 38 publications

Convergence Rate Analysis of the Forward-Douglas-Rachford Splitting Scheme

Convergence Rate Analysis of the Forward-Douglas-Rachford Splitting Scheme

Splitting with Near-Circulant Linear Systems: Applications to Total Variation CT and PET

Proximal alternating penalty algorithms for nonsmooth constrained convex optimization

Contact Info

Product

Resources

About