Structured nonconvex and nonsmooth optimization: algorithms and iteration complexity analysis

Abstract

Nonconvex and nonsmooth optimization problems are frequently encountered in much of statistics, business, science and engineering, but they are not yet widely recognized as a technology in the sense of scalability. A reason for this relatively low degree of popularity is the lack of a well developed system of theory and algorithms to support the applications, as is the case for its convex counterpart. This paper aims to take one step in the direction of disciplined nonconvex and nonsmooth optimization. In particular, we consider in this paper some constrained nonconvex optimization models in block decision variables, with or without coupled affine constraints. In the absence of coupled constraints, we show a sublinear rate of convergence to an ǫ-stationary solution in the form of variational inequality for a generalized conditional gradient method, where the convergence rate is dependent on the Hölderian continuity of the gradient of the smooth part of the objective. For the model with coupled affine constraints, we introduce corresponding ǫ-stationarity conditions, and apply two proximal-type variants of the ADMM to solve such a model, assuming the proximal ADMM updates can be implemented for all the block variables except for the last block, for which either a gradient step or a majorization-minimization step is implemented. We show an iteration complexity bound of O(1/ǫ 2 ) to reach an ǫ-stationary solution for both algorithms. Moreover, we show that the same iteration complexity of a proximal BCD method follows immediately. Numerical results are provided to illustrate the efficacy of the proposed algorithms for tensor robust PCA.

Efficient Optimization Algorithms for Robust Principal Component Analysis and Its Variants

“…Problem ǫ-optimality measure Convergence rate Algorithms for Convex Models PGM [16] (4) objective value error O(1/k) APGM [16] (4) objective value error O(1/k 2 ) IALM [17] (2) -convergence, no rate given ADMM [18] (2) -convergence, no rate given ALM [19] (17) objective value error O(1/k) FALM [19] (17) objective value error O(1/k 2 ) ASALM [20] (19) -convergence unclear, no rate given VASALM [20] (19) -convergence, no rate given PSPG [21] (3) objective value error O(1/k) ADMIP [22] (3) objective value error O(1/k) Quasi-Newton method (fastRPCA) [23] (26) -convergence, no rate given 3-block ADMM [24] (28) -convergence, no rate given Frank-Wolfe [25] (30) objective value error O(1/k) Algorithms for Nonconvex Models GoDec [26] (33) -local convergence, no rate given GreBsmo [27] (36) -convergence unclear, no rate given Alternating Minimization (R2PCP) [28] (35) -local convergence, no rate given Gradient Descent (GD) [11] ≈ (36) -linear convergence with proper initialization and incoherence assumption Alternating Minimization [29] (37) -local convergence with proper initialization and incoherence and RIP assumptions Stochastic alg. [30] (39) -convergence if the iterates are always full rank matrices, no rate given LMafit [31] (44) -convergence if difference between two consecutive iterates tends to zero, no rate given Conditional Gradient [32] (48) [32] (54) perturbed KKT O(1/ √ k) † Note: Some of these algorithms solve different problems and the ǫ-optimality measures are also different, so the convergence rates are not directly comparable with each other. [11] has no explicit optimization formulation but the objective is similar to (36).…”

An Efficient ADMM-Based Algorithm to Nonconvex Penalized Support Vector Machines

“…Apart from previous work discussed above, many researches concerned with optimization problems with a general class of nonconvex regularizations [12], [13], [19], [20] are developed. Nevertheless, these proposed methods cannot be applied to solve the optimization problem studied in this paper.…”

“…Gong et al [12] proposed General Iterative Shrinkage and Thresholding (GIST) algorithm to solve the nonconvex optimization problem for a large class of nonconvex penalties. Recently, Jiang et al [13] have proposed two proximal-type variants of the ADMM to solve the structured nonconvex and nonsmooth problems. Nevertheless, the algorithms proposed in [12] and [13] are unable to solve the nonconvex penalized hinge loss function because they both require the loss function to be differentiable as well.…”

“…Consequently, equation (13) can be equivalently converted to (17) is O(dn 2 ) flops, giving rise to an improvement by a factor of (d/n) 2 over the naive w-update by Equation (13).…”

“…Existing solutions to nonconvex penalized SVMs [2], [11] are pretty computationally inefficient, and they are limited to a few of nonconvex penalties. Besides that, other popular approaches are unable to apply to the nonconvex penalized hinge loss function since they typically require the loss function to be differentiable [12], [13].…”