A second-order method for convex₁-regularized optimization with active-set prediction

Abstract

We describe an active-set method for the minimization of an objective function φ that is the sum of a smooth convex function f and an 1 -regularization term. A distinctive feature of the method is the way in which active-set identification and second-order subspace minimization steps are integrated to combine the predictive power of the two approaches. At every iteration, the algorithm selects a candidate set of free and fixed variables, performs an (inexact) subspace phase, and then assesses the quality of the new active set. If it is not judged to be acceptable, then the set of free variables is restricted and a new active-set prediction is made. We establish global convergence for our approach under the assumptions of Lipschitz-continuity and strong-convexity of f, and compare the new method against state-of-the-art codes.

A Highly Efficient Semismooth Newton Augmented Lagrangian Method for Solving Lasso Problems

“…Since U ∈ ℜ n×n is a diagonal matrix, at the first glance, the costs of computing AU A T and the matrixvector multiplication AU A T d for a given vector d ∈ ℜ m are O(m 2 n) and O(mn), respectively. These computational costs are too expensive when the dimensions of A are large and can make the commonly employed approaches such as the Cholesky factorization and the conjugate gradient method inappropriate for solving (26). Fortunately, under the sparse optimization setting, if the sparsity of U is wisely taken into the consideration, one can substantially reduce these unfavorable computational costs to a level such that they are negligible or at least insignificant compared to other costs.…”

“…See Figure 2 for an illustration on the computation of A T J A J . In this case, the total computational costs for solving the Newton linear system (26) are reduced significantly further from O(m 2 (m + r)) to O(r 2 (m + r)). We should emphasize here that this dramatic reduction on the computational costs results from the wise combination of the careful examination of the existing second order sparsity in the Lasso-type problems and some "smart" numerical linear algebra.…”

Subspace Methods for Nonlinear Optimization

“…The active set strategies have also been studied in [65,107]. Specifically, the method in [65] solves a smooth quadratic subproblem determined by the active sets and invokes a corrective cycle that greatly improves the efficiency and robustness of the algorithm. The method is globalized by using a proximal gradient step to check the desired progress.…”

“…A subset of the variables is fixed in the so-called active sets determined by certain mechanisms and the remaining variables are computed from carefully constructed subproblems. Examples include optimization problems with bound constraints or linear constraints in [17,18,53,82,83], ℓ 1 -regularized problem for sparse optimization in [65,107,135] and general nonlinear programs in [19,20]. In quadratic programming, the inequality constraints that have zero values at the optimal solution are called active, and they are replaced by equality constraints in the subproblem [113].…”

A Fast, Principled Working Set Algorithm for Exploiting Piecewise Linear Structure in Convex Problems

“…These are just two of many algorithms that incorporate working sets to speed up sparse optimization. For lasso-type problems, many additional studies combine working set (Scheinberg and Tang, 2016;Massias et al, 2017) or active set (Wen et al, 2012;Solntsev et al, 2015;Keskar et al) strategies with standard algorithms. Researchers have also applied working sets to many other sparse problems-see e.g.…”