Parallel Multi-Block ADMM with o(1 / k) Convergence

Abstract

Abstract. This paper introduces a parallel and distributed extension to the alternating direction method of multipliers (ADMM) for solving convex problem:The algorithm decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This Jacobian-type algorithm is well suited for distributed computing and is particularly attractive for solving certain large-scale problems.This paper introduces a few novel results. Firstly, it shows that extending ADMM straightforwardly from the classic Gauss-Seidel setting to the Jacobian setting, from 2 blocks to N blocks, will preserve convergence if matrices Ai are mutually near-orthogonal and have full column-rank. Secondly, for general matrices Ai, this paper proposes to add proximal terms of different kinds to the N subproblems so that the subproblems can be solved in flexible and efficient ways and the algorithm converges globally at a rate of o(1/k). Thirdly, a simple technique is introduced to improve some existing convergence rates from O(1/k) to o(1/k).In practice, some conditions in our convergence theorems are conservative. Therefore, we introduce a strategy for dynamically tuning the parameters in the algorithm, leading to substantial acceleration of the convergence in practice. Numerical results are presented to demonstrate the efficiency of the proposed method in comparison with several existing parallel algorithms.We implemented our algorithm on Amazon EC2, an on-demand public computing cloud, and report its performance on very large-scale basis pursuit problems with distributed data.

Convergence Analysis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problems

“…For the multi-block separable convex problems where K ≥ 3, it is known that the original ADMM can diverge for certain pathological problems [26]. Therefore, most research effort in this direction has been focused on either analyzing problems with additional conditions, or showing convergence for variants of the ADMM; see for example [26][27][28][29][30][31][32][33][34]. It is worth mentioning that when the objective function is not separable across the variables (e.g., the coupling function ℓ(·) appears in the objective), the convergence of the ADMM is still open, even in the case where K = 2 and f (·) is convex.…”

A Proximal Fully Parallel Splitting Method for Stable Principal Component Pursuit

“…We compare it with the splitting method for separable convex programming (denoted by HTY) in [30], the full Jacobian decomposition of the augmented Lagrangian method (denoted by FJDALM) in [13], and the fully parallel ADMM (denoted by PADMM) in [14]. Through these numerical experiments, we want to illustrate that (1) suitable proximal terms can enhance PFPSM's numerical efficiency in practice, (2) larger values of the Glowinski relaxation factor can often accelerate PFPSM's convergence speed, and (3) compared with the other three ADMM-type methods the dynamically updated step size defined in (27) can accelerate the convergence of PFPSM.…”

“…Compared to the iteration method in [13], the new iteration method PFPSM extends the scope of from 1 to the interval ( √ 3/2, 2/ √ 3), and larger values of are often more preferred in practice; see Figure 2. Furthermore, compared to the iteration method in [14], the new iteration method PFPSM removes the restriction imposed on the regularized matrices ( = 1, 2, 3); see condition (2.10) in [14].…”

“…Recently, there has been an intensive study on the iteration methods for solving (6) [5,6,[10][11][12]. Interestingly, these methods usually only exploit the first-order information of the objective function of (6), and most of them are based on the well-known alternating direction method with multipliers (ADMM), such as the sequential ADMM [11], the partially parallel ADMMs [5,6,10,12], and the fully parallel ADMMs [13,14]. As an influential first-order iteration method in optimization, ADMM was independently proposed by Glowinski and Marrocco [15] and Gabay and Mercier [16] about forty years ago, and due to its high efficiency in the numerical simulation, ADMM has been receiving considerable attention in recent years.…”

“…Chen et al [20] showed that the sequential ADMM cannot directly extend to three-block separable convex programming by a simple counterexample, and similarly He et al [13] showed that the full parallel ADMM may be divergent even for two-block separable convex programming. There are three strategies to deal with the issue: (1) the substitution procedure [13,21]; (2) the technique of regularization [14,22]; (3) the technique of small step size [23]. For other recent developments of ADMM, including the (sub)linear convergence rate, various acceleration techniques, indefinite regularized matrices, larger step size, and its generalization to solve nonconvex and nonsmooth multiblock separable programming, we refer to [24][25][26][27][28].…”