Global Convergence of Unmodified 3-Block ADMM for a Class of Convex Minimization Problems

Nonconvex and nonsmooth optimization problems are frequently encountered in much of statistics, business, science and engineering, but they are not yet widely recognized as a technology in the sense of scalability. A reason for this relatively low degree of popularity is the lack of a well developed system of theory and algorithms to support the applications, as is the case for its convex counterpart. This paper aims to take one step in the direction of disciplined nonconvex and nonsmooth optimization. In particular, we consider in this paper some constrained nonconvex optimization models in block decision variables, with or without coupled affine constraints. In the absence of coupled constraints, we show a sublinear rate of convergence to an ǫ-stationary solution in the form of variational inequality for a generalized conditional gradient method, where the convergence rate is dependent on the Hölderian continuity of the gradient of the smooth part of the objective. For the model with coupled affine constraints, we introduce corresponding ǫ-stationarity conditions, and apply two proximal-type variants of the ADMM to solve such a model, assuming the proximal ADMM updates can be implemented for all the block variables except for the last block, for which either a gradient step or a majorization-minimization step is implemented. We show an iteration complexity bound of O(1/ǫ 2 ) to reach an ǫ-stationary solution for both algorithms. Moreover, we show that the same iteration complexity of a proximal BCD method follows immediately. Numerical results are provided to illustrate the efficacy of the proposed algorithms for tensor robust PCA.

show abstract

“…For applications of the sharing problem, see [13,34,40,41]. Our proximal ADMM-g for solving (1.1) with A N = I is described in Algorithm 2.…”

Section: Proximal Gradient-based Admm (Proximal Admm-g)mentioning

confidence: 99%

Structured nonconvex and nonsmooth optimization: algorithms and iteration complexity analysis

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…Proof : See Appendix A. Once θ t+1 is obtained, updates (17c) and (17d) can be performed in parallel and we have effectively transformed a 3-block ADMM in (11) to a 2-block ADMM, which converges under more general settings [20].…”

Section: Algorithmic Description and Implementationmentioning

confidence: 99%

“…As it can be seen in Figure 1 and 2, the BFGS-ADMM method demonstrates significant speed up in both datasets compared to other methods. Note that our first order variant also shows advantages compared to other first order methods, due to the fact that in our first order approximation, step size selection also takes into account the number of neighbors each agent has as in (20).…”

Section: Convergence Analysismentioning

confidence: 99%

BFGS-ADMM for Large-Scale Distributed Optimization

Li¹,

Gong²,

Freris³

et al. 2021

Preprint

View full text Add to dashboard Cite

We consider a class of distributed optimization problem where the objective function consists of a sum of strongly convex and smooth functions and a (possibly nonsmooth) convex regularizer. A multi-agent network is assumed, where each agent holds a private cost function and cooperates with its neighbors to compute the optimum of the aggregate objective. We propose a quasi-Newton Alternating Direction Method of Multipliers (ADMM) where the primal update is solved inexactly with approximated curvature information. By introducing an intermediate consensus variable, we achieve a block diagonal Hessian which eliminates the need for inner communication loops within the network when computing the update direction. We establish global linear convergence to the optimal primal-dual solution without the need for backtracking line search, under the assumption that component cost functions are strongly convex with Lipschitz continuous gradients. Numerical simulations on real datasets demonstrate the advantages of the proposed method over state of the art.

show abstract

“…The global convergence of IALM when applied for optimizing a convex problem with at most two blocks has been theoretically proven [40], [45], [46]. However, there are three blocks in Algorithm 1, and to the best of our knowledge, the theoretical global convergence of the IALM solver with three or more blocks is still unsolved [45].…”

Section: Discriminative Low-rank Representation 1) Problem Formulationmentioning

confidence: 99%

Superpixel-guided Discriminative Low-rank Representation of Hyperspectral Images for Classification

Yang,

Hou,

Jia

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we propose a novel classification scheme for the remotely sensed hyperspectral image (HSI), namely SP-DLRR, by comprehensively exploring its unique characteristics, including the local spatial information and lowrankness. SP-DLRR is mainly composed of two modules, i.e., the classification-guided superpixel segmentation and the discriminative low-rank representation, which are iteratively conducted. Specifically, by utilizing the local spatial information and incorporating the predictions from a typical classifier, the first module segments pixels of an input HSI (or its restoration generated by the second module) into superpixels. According to the resulting superpixels, the pixels of the input HSI are then grouped into clusters and fed into our novel discriminative lowrank representation model with an effective numerical solution. Such a model is capable of increasing the intra-class similarity by suppressing the spectral variations locally while promoting the inter-class discriminability globally, leading to a restored HSI with more discriminative pixels. Experimental results on three benchmark datasets demonstrate the significant superiority of SP-DLRR over state-of-the-art methods, especially for the case with an extremely limited number of training pixels.

show abstract

Global Convergence of Unmodified 3-Block ADMM for a Class of Convex Minimization Problems

Cited by 43 publications

References 40 publications

Structured nonconvex and nonsmooth optimization: algorithms and iteration complexity analysis

Structured nonconvex and nonsmooth optimization: algorithms and iteration complexity analysis

BFGS-ADMM for Large-Scale Distributed Optimization

Superpixel-guided Discriminative Low-rank Representation of Hyperspectral Images for Classification

Contact Info

Product

Resources

About