Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions

Davis, Damek; Yin, Wotao

doi:10.1287/moor.2016.0827

Cited by 90 publications

(151 citation statements)

References 38 publications

(111 reference statements)

Supporting

Mentioning

144

Contrasting

Order By: Relevance

“…Thus, this example shows that the assumption "T −1 is Lipschitz continuity at 0" is weaker than the strong monotonicity assumption on T as assumed in [8,9,15].…”

mentioning

confidence: 88%

“…Note that, as analyzed in [33], "the assumption of Lipschitz continuity of T −1 at 0 turns out to be very natural in applications to convex programming". Indeed, we will show later that this assumption is weaker than those considered in [8,9,15] (see the example in Section 2.2) and it suffices to ensure the linear convergence of the schemes (1.7) and (1.8) for the case γ ∈ (0, 2). Thus, the distinction of this work from existing results in the literature is that stronger convergence rates are established under weaker conditions for the generalized PPA schemes (1.7) and (1.8).…”

mentioning

confidence: 99%

See 1 more Smart Citation

On the Optimal Linear Convergence Rate of a Generalized Proximal Point Algorithm

Tao

Yuan

2017

J Sci Comput

View full text Add to dashboard Cite

Abstract. The proximal point algorithm (PPA) has been well studied in the literature. In particular, its linear convergence rate has been studied by Rockafellar in 1976 under certain condition. We consider a generalized PPA in the generic setting of finding a zero point of a maximal monotone operator, and show that the condition proposed by Rockafellar can also sufficiently ensure the linear convergence rate for this generalized PPA. Indeed we show that these linear convergence rates are optimal.Both the exact and inexact versions of this generalized PPA are discussed. The motivation to consider this generalized PPA is that it includes as special cases the relaxed versions of some splitting methods that are originated from PPA. Thus, linear convergence results of this generalized PPA can be used to better understand the convergence of some widely used algorithms in the literature. We focus on the particular convex minimization context and specify Rockafellar's condition to see how to ensure the linear convergence rate for some efficient numerical schemes, including the classical augmented Lagrangian method proposed by Hensen and Powell in 1969 and its relaxed version, the original alternating direction method of multipliers (ADMM) by Glowinski and Marrocco in 1975 and its relaxed version (i.e., the generalized ADMM by Eckstein and Bertsekas in 1992).Some refined conditions weaker than existing ones are proposed in these particular contexts.

show abstract

“…Thus, this example shows that the assumption "T −1 is Lipschitz continuity at 0" is weaker than the strong monotonicity assumption on T as assumed in [8,9,15].…”

mentioning

confidence: 88%

mentioning

confidence: 99%

On the Optimal Linear Convergence Rate of a Generalized Proximal Point Algorithm

Tao

Yuan

2017

J Sci Comput

View full text Add to dashboard Cite

show abstract

“…where α J is defined as (20). If additionally λ ν + λ 1−γ 2 ℓ 2 ≥ 0 whenever γℓ < 1, then the Lipschitz constant (31) can be improved to…”

Section: Lemma 36 (Resolvents Of Lipschitz α-Monotone Operators) Letmentioning

confidence: 99%

Adaptive Douglas--Rachford Splitting Algorithm for the Sum of Two Operators

Dao¹,

Phan²

2019

SIAM J. Optim.

View full text Add to dashboard Cite

The Douglas-Rachford algorithm is a classical and powerful splitting method for minimizing the sum of two convex functions and, more generally, finding a zero of the sum of two maximally monotone operators. Although this algorithm has been well understood when the involved operators are monotone or strongly monotone, the convergence theory for weakly monotone settings is far from being complete. In this paper, we propose an adaptive Douglas-Rachford splitting algorithm for the sum of two operators, one of which is strongly monotone while the other one is weakly monotone. With appropriately chosen parameters, the algorithm converges globally to a fixed point from which we derive a solution of the problem. When one operator is Lipschitz continuous, we prove global linear convergence which sharpens recent known results. entire mathematical model. It is worth mentioning (see, e.g., [23]) that several splitting methods such as the method of partial inverses [35] and the alternating direction method of multipliers (ADMM) [25] can be written in the form of the DR algorithm, which itself can be transformed into the proximal point algorithm [34]. Other splitting schemes can be found in [13,14,16] and the references therein.When applied to two normal cone operators, the DR algorithm can be used to solve the feasibility problem of finding a common point of two sets. In this context, the DR algorithm possesses many good properties, for example, it finds a best approximation point when the intersection of sets is empty [3,5,8], it finds an exact solution after only a finite number of iterations under verifiable conditions [1,4,7], and it converges globally in some nonconvex settings [9,19] while converges locally with linear or sublinear rate under some regularity assumptions [11,29,33]. In the absence of constraint qualifications, [6] suggests that the DR algorithm outperforms the well-known method of alternating projections. In attempting to generalize the DR algorithm for feasibility problems, several parameters were added to its formulation [12,18,17,24]. In this case, one has the freedom to modify the parameters that are associated with the projections without giving up the solution. This approach is possible because the underlying normal cone operators have homogeneous values, which allows for scaling them independently. The situation changes completely when working with general problems where two involved operators may no longer have such homogeneity. In this case, a naive scaling may destroy the ability to solve the original problem. Therefore, we aim to overcome this hurdle by proposing an adaptive approach.The paper is devoted to the convergence analysis of the adaptive DR algorithm for finding a zero of the sum of α-and β-monotone operators, in which α-monotonicity is a unification of strong and weak monotonicity (see Definition 3.1). This situation arises in various important applications; see [27] for a brief discussion. The main contribution is summarized below.(R1) We incorporate parameters into the DR algorith...

show abstract

“…Our analysis builds on the techniques and results of [7,16,17]. The rest of this section contains a brief review of these results.…”

Section: Introductionmentioning

confidence: 99%

“…Furthermore, it is unclear how the FDRS algorithm relates to other algorithms. We seek to fill this gap.The techniques used in this paper are based on [15,16,17]. These techniques are quite different from those used in classical objective error convergence rate analysis.…”

mentioning

confidence: 99%

Convergence Rate Analysis of the Forward-Douglas-Rachford Splitting Scheme

Davis¹

2015

SIAM J. Optim.

Self Cite

View full text Add to dashboard Cite

Operator splitting schemes are a class of powerful algorithms that solve complicated monotone inclusion and convex optimization problems that are built from many simpler pieces. They give rise to algorithms in which all simple pieces of the decomposition are processed individually. This leads to easily implementable and highly parallelizable or distributed algorithms, which often obtain nearly state-of-the-art performance. In this paper, we analyze the convergence rate of the forwardDouglas-Rachford splitting (FDRS) algorithm, which is a generalization of the forward-backward splitting (FBS) and Douglas-Rachford splitting algorithms. Under general convexity assumptions, we derive the ergodic and nonergodic convergence rates of the FDRS algorithm, and show that these rates are the best possible. Under Lipschitz differentiability assumptions, we show that the best iterate of FDRS converges as quickly as the last iterate of the FBS algorithm. Under strong convexity assumptions, we derive convergence rates for a sequence that strongly converges to a minimizer. Under strong convexity and Lipschitz differentiability assumptions, we show that FDRS converges linearly. We also provide examples where the objective is strongly convex, yet FDRS converges arbitrarily slowly. Finally, we relate the FDRS algorithm to a primal-dual FBS scheme and clarify its place among existing splitting methods. Our results show that the FDRS algorithm automatically adapts to the regularity of the objective functions and achieves rates that improve upon the sharp worst case rates that hold in the absence of smoothness and strong convexity. 1761The forward-backward splitting (FBS) algorithm [23] is another technique for solving (1.1) when g is known to be smooth. In this case, the proximal operator of g is never evaluated. Instead, FBS combines gradient (forward) steps with respect to g and proximal (backward) steps with respect to f . FBS is especially useful when the proximal operator of g is complex and its gradient is simple to compute.Recently, the forward-Douglas-Rachford splitting (FDRS) algorithm [7] was proposed to combine DRS and FBS and extend their applicability (see Algorithm 1). More specifically, let V ⊆ H be a closed vector space and suppose g is smooth. Then FDRS applies to the following constrained problem:Problem (1.5) arises in the dual form soft-margin kernelized support vector machine classifier [14] in which C is a box constraint, b is 0, and A has rank one. Note that by the argument in (1.3), we can always assume that b = 0. Define the smooth function g(x) := (1/2) Qx, x + c, x , the indicator function f (x) := χ C (x) (which is 0 on C and ∞ elsewhere), and the vector space V := {x ∈ R d | Ax = 0}. With this notation, (1.5) is in the form (1.2) and, thus, FDRS can be applied. This splitting is nice because ∇g(x) = Qx + c is simple whereas the proximal operator of g requires a matrix inversion prox γg = (I R d +γQ) −1 •(I R d −γc), which is expensive for large-scale problems. Goals, challenges, and approaches.This work se...

show abstract

Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions

Cited by 90 publications

References 38 publications

On the Optimal Linear Convergence Rate of a Generalized Proximal Point Algorithm

On the Optimal Linear Convergence Rate of a Generalized Proximal Point Algorithm

Adaptive Douglas--Rachford Splitting Algorithm for the Sum of Two Operators

Convergence Rate Analysis of the Forward-Douglas-Rachford Splitting Scheme

Contact Info

Product

Resources

About