On the linear convergence of the alternating direction method of multipliers

Abstract

We analyze the convergence rate of the alternating direction method of multipliers (ADMM) for minimizing the sum of two or more nonsmooth convex separable functions subject to linear constraints. Previous analysis of the ADMM typically assumes that the objective function is the sum of only two convex functions defined on two separable blocks of variables even though the algorithm works well in numerical experiments for three or more blocks. Moreover, there has been no rate of convergence analysis for the ADMM without strong convexity in the objective function. In this paper we establish the global linear convergence of the ADMM for minimizing the sum of any number of convex separable functions. This result settles a key question regarding the convergence of the ADMM when the number of blocks is more than two or if the strong convexity is absent. It also implies the linear convergence of the ADMM for several contemporary applications including LASSO, Group LASSO and Sparse Group LASSO without any strong convexity assumption. Our proof is based on estimating the distance from a dual feasible solution to the optimal dual solution set by the norm of a certain proximal residual, and by requiring the dual stepsize to be sufficiently small.KEY WORDS: Linear convergence, alternating directions of multipliers, error bound, dual ascent.

Directed Acyclic Graph Continuous Max-Flow Image Segmentation for Unconstrained Label Orderings

“…The convergence of said augmented Lagrangian algorithm is on the order of O(1/N ) similar to other primal-dual approaches [12]. This was shown by Hong and Luo [25] for multi-block alternating direction method of multipliers algorithms such as our algorithm. This convergence relies on the decomposition of cost function and constraints across each block having particular, restrictive properties which are upheld for all inter-node flows,…”

“…This guarantees linear convergence under weaker restrictions [25] and provides a theoretical justification for the use of only a single Chambolle iteration per update step.…”

A survey on operator splitting and decomposition of convex programs

“…It appears that linear convergence is possible when at least one of the block function is strongly convex. A recent study by Hong and Luo [48] made new progress in relaxing these assumptions and extending too to more than two blocks. Their separable model extends the M-model to include the S-model used in the former section.…”

Distributed Online Modified Greedy Algorithm for Networked Storage Operation Under Uncertainty

“…Multi-block variants of ADMM may eliminate such need. However, the convergence is not guaranteed or requires additional assumptions [30], [31], [32]. Validating these assumptions for specific storage control problem instances may lead to simpler algorithm which has similar convergence properties.…”