Hybrid Jacobian and Gauss--Seidel Proximal Block Coordinate Update Methods for Linearly Constrained Convex Programming

Abstract

Recent years have witnessed the rapid development of block coordinate update (BCU) methods, which are particularly suitable for problems involving large-sized data and/or variables. In optimization, BCU first appears as the coordinate descent method that works well for smooth problems or those with separable nonsmooth terms and/or separable constraints. As nonseparable constraints exist, BCU can be applied under primal-dual settings.In the literature, it has been shown that for weakly convex problems with nonseparable linear constraint, BCU with fully Gauss-Seidel updating rule may fail to converge and that with fully Jacobian rule can converge sublinearly. However, empirically the method with Jacobian update is usually slower than that with Gauss-Seidel rule. To maintain their advantages, we propose a hybrid Jacobian and Gauss-Seidel BCU method for solving linearly constrained multi-block structured convex programming, where the objective may have a nonseparable quadratic term and separable nonsmooth terms. At each primal block variable update, the method approximates the augmented Lagrangian function at an affine combination of the previous two iterates, and the affinely mixing matrix with desired nice properties can be chosen through solving a semidefinite programming. We show that the hybrid method enjoys the theoretical convergence guarantee as Jacobian BCU. In addition, we numerically demonstrate that the method can perform as well as Gauss-Seidel method and better than a recently proposed randomized primal-dual BCU method.

CoordinateWise Descent Methods for Leading Eigenvalue Problem

“…Readers are referred to the recent surveys [35,47] and references therein for detailed history of the development and analysis of general CDMs. Since the survey papers, the area of CDMs is still under fast development, see e.g., [15,33,48,51]. Momentum acceleration can also be combined with the CDMs to further accelerate the convergence [2,20,22,29].…”

Accelerated primal–dual proximal block coordinate updating methods for constrained convex optimization

“…[4,6,16,25,26]) such as strong convexity or orthogonality conditions on the linear constraint. Without additional assumptions, modification is necessary for the ADMM applied to multi-block problems to be convergent; see [12,19,20,39] for example. Very recently, [15] proposed a randomized primal-dual coordinate (RPDC) update method, whose asynchronous parallel version was then studied in [41].…”

“…Assume that B is full row-rank and max{ A 2 , B 2 } ≤ 1. Choose δ, κ, η x , η y , c, β, τ 1 , τ 2 satisfying (38) and (39), and in addition,…”

First-Order Methods for Constrained Convex Programming Based on Linearized Augmented Lagrangian Function

“…First-order BCU methods have been extensively studied for problems with separable or affine constraints (Tseng and Yun 2009, Nesterov 2012, Razaviyayn et al 2013, Xu and Yin 2013, Gao and Zhang 2017, Xu 2018, Xu and Zhang 2018. This work appears, to the best of the author's knowledge, to be the first on BCUs for nonlinear inequality constrained problems.…”