Convergence Rate Analysis of Several Splitting Schemes

Abstract

Splitting schemes are a class of powerful algorithms that solve complicated monotone inclusions and convex optimization problems that are built from many simpler pieces. They give rise to algorithms in which the simple pieces of the decomposition are processed individually. This leads to easily implementable and highly parallelizable algorithms, which often obtain nearly state-of-the-art performance.In the first part of this paper, we analyze the convergence rates of several general splitting algorithms and provide examples to prove the tightness of our results. The most general rates are proved for the fixed-point residual (FPR) of the Krasnosel'skiȋ-Mann (KM) iteration of nonexpansive operators, where we improve the known big-O rate to little-o. We show the tightness of this result and improve it in several special cases. In the second part of this paper, we use the convergence rates derived for the KM iteration to analyze the objective error convergence rates for the Douglas-Rachford (DRS), Peaceman-Rachford (PRS), and ADMM splitting algorithms under general convexity assumptions. We show, by way of example, that the rates obtained for these algorithms are tight in all cases and obtain the surprising statement: The DRS algorithm is nearly as fast as the proximal point algorithm (PPA) in the ergodic sense and nearly as slow as the subgradient method in the nonergodic sense. Finally, we provide several applications of our result to feasibility problems, model fitting, and distributed optimization. Our analysis is self-contained, and most results are deduced from a basic lemma that derives convergence rates for summable sequences, a simple diagram that decomposes each relaxed PRS iteration, and fundamental inequalities that relate the FPR to objective error.Keywords Krasnosel'skiȋ-Mann algorithm · Douglas-Rachford Splitting · Peaceman-Rachford Splitting · Alternating Direction Method of Multipliers · nonexpansive operator · averaged operator · fixed-point algorithm · little-o convergence Mathematics Subject Classification (2000) 47H05 · 65K05 · 65K15 · 90C25

Convergence Analysis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problems

“…In this case, ADMM is known to converge under very mild conditions; see [7] and [8]. Under the same conditions, several recent works [20][21][22] have shown that the ADMM converges with the sublinear rate of O( …”

A survey on operator splitting and decomposition of convex programs

“…Moreover, it was shown in [26] that the sequence x t − x * is nonincreasing and that Sx t − x t 2 = o(1/t), assuming only that the sequence τ t = ρ t (1 − ρ t ) is bounded away from 0. Conveniently, compositions of averaged operators are easily seen to be averaged.…”

“…On the other hand, most users of (DRA) have fixed γ = 1 to focus on the estimation of the scaling parameter λ as seen below. More insight on relaxed versions of (FB), (DRA) and (PRA) and their theoretical rates of convergence may be found in the recent study by Davis and Yin and companion papers [26].…”

“…Some results may look rather frustrating as commented in a recent study [26] : the global convergence of Douglas-Rachford splitting scheme can be as fast as the Proximal iteration in the ergodic sense and as slow as a subgradient method in the non ergodic sense.…”

The augmented Lagrangian method with full Jacobian decomposition and logarithmic-quadratic proximal regularization for multiple-block separable convex programming

“…However, it was shown in [6] that the scheme (1.6) is not necessarily convergent. The convergence rate of ADMM and its extension are analysed in [9,28,30,33,32].…”