Convergence Analysis of L-ADMM for Multi-block Linear-Constrained Separable Convex Minimization Problem

Abstract

We focus on the convergence analysis of the extended linearized alternating direction method of multipliers (L-ADMM) for solving convex minimization problems with three or more separable blocks in the objective functions. Previous convergence analysis of the L-ADMM needs to reduce the multi-block convex minimization problems to two blocks by grouping the variables. Moreover, there has been no rate of convergence analysis for the L-ADMM. In this paper, we construct a counter example to show the failure of convergence of the extended L-ADMM. We prove the convergence and establish the sublinear convergence rate of the extended L-ADMM under the assumptions that the proximal gradient step sizes are smaller than certain values, and any two coefficient matrices in linear constraints are orthogonal.

The linear convergence rate of the linearized version of alternating direction method of multipliers for convex optimization with three separable functions

“…The global convergence of the iterative sequence (1.5) is further proved (see [8]). Under strongly convex conditions, Yao and Cheng obtained the global convergence of the above algorithm (1.5) under the condition of 3 m  (see [9]).…”

“…) [7] proves that the algorithm is convergent. In [8], Feng et al directly extended the linearized alternating direction multiplier method in the iterative algorithm (1.4) to the case with three separable operators to solve the problem (1.1). The iterative sequence (DLADMM) in this direct extension case is written as the following iterative form…”

“…In fact, if there is no further assumption, the global convergence of the above algorithm (1.5) does not hold. Feng et al [8] constructs a sufficient condition to ensure the convergence of L-ADMM (1.5) with multi-blocks cases, that is, the existence of any two coefficient matrices in problem (1.1) is orthogonal and max ( )…”

Hybrid Jacobian and Gauss--Seidel Proximal Block Coordinate Update Methods for Linearly Constrained Convex Programming

“…However, the Gauss-Seidel update requires additional assumptions for convergence, though it can empirically perform better than the Jacobi method if it happens to converge. Counterexamples are constructed in [4,12] to show possible divergence of Gauss-Seidel block coordinate update for linearly constrained problems. To guarantee convergence of the Gauss-Seidel update, many existing works assume strong convexity on the objective or part of it.…”

“…Empirically, the latter one usually performs better. However, theoretically, the latter case may fail to converge when m ≥ 3, as shown in [4,12]. Therefore, we will design a mixing matrix W between the above two choices to theoretically guarantee convergence and also maintain practically nice performance.…”