A family of second-order methods for convex $$\ell _1$$-regularized optimization

Byrd, Richard H.; Chin, Gillian M.; Nocedal, Jorge; Oztoprak, Figen

doi:10.1007/s10107-015-0965-3

Cited by 43 publications

(46 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…OWL-QN needs only matrixvector multiplications since it use L-BFGS [20] method to form the inverse of the Hessian matrix. Although OWL-QN has been proved very effective in practice, but no convergence analysis was provided [4,22,29]. Pinghua Gong and Jieping Ye [13] proposed a modified Orthant-Wise Limited Memory Quasi-Newton Method (mOWL-QN), which establish a detailed convergence analysis for the OWK-QN-type algorithm and also have similarly convergence as the OWL-QN.…”

Section: Introductionmentioning

confidence: 99%

Inexact Orthant-Wise Quasi-Newton Method

Yao²,

Zhang³

et al. 2018

Proceedings of the 2018 2nd International Conference on Applied Mathematics, Modelling and Statistics Application (AMMSA 2018)

View full text Add to dashboard Cite

Abstract-The Orthant-Wise Limited-memory QuasiNewton method (OWL-QN), based on the L-BFGS method, is an effective algorithm for solving the '1-regularized sparse learning problem. In order to deal with the '1-regularization, OWL-QN restrict the point to an orthant on which the quadratic model is valid and differentiable. In this paper, we propose an Inexact Orthant-Wise Limited-memory Quasi-Newton method (IOWL-QN). This method, at every iteration, compute an approximate solution satisfied the inexactness conditions to estimate the exact solution. We give brief proof to the convergence and report the numerical results.

show abstract

Section: Introductionmentioning

confidence: 99%

Inexact Orthant-Wise Quasi-Newton Method

Yao²,

Zhang³

et al. 2018

Proceedings of the 2018 2nd International Conference on Applied Mathematics, Modelling and Statistics Application (AMMSA 2018)

View full text Add to dashboard Cite

show abstract

“…More recently, some authors have investigated the use of second order information in cases where the objective function is composed by a regular function and the 1 -norm of the design variable [2,4,6]. By using second order information of the regular part, faster algorithms have been obtained.…”

Section: Introductionmentioning

confidence: 99%

“…In the pioneering work [2], a limited memory BFGS method was considered in connection with so-called orthantwise directions. In [4] and [6] these type of directions were considered in connection with Newton and semismooth Newton type updates, respectively. Active set strategies based on second order information have also been recently envisaged ( [31,34,40]).…”

Section: Introductionmentioning

confidence: 99%

“…Byrd et al [6] considered this framework for the solution of problem (P). By appropriately choosing the algorithm parameters, several well-known algorithms (FISTA [3], OWL [2]) may be derived from this rather general setting.…”

mentioning

confidence: 99%

“…Let us start by recalling some basic notions of semismooth Newton methods and the general framework proposed in [6].…”

mentioning

confidence: 99%

See 2 more Smart Citations

Second-order orthant-based methods with enriched Hessian information for sparse $$\ell _1$$ ℓ 1 -optimization

2017

View full text Add to dashboard Cite

Abstract. We present a second order algorithm, based on orthantwise directions, for solving optimization problems involving the sparsity enhancing 1-norm. The main idea of our method consists in modifying the descent orthantwise directions by using second order information both of the regular term and (in weak sense) of the 1-norm. The weak second order information behind the 1-term is incorporated via a partial Huber regularization. One of the main features of our algorithm consists in a faster identification of the active set. We also prove that a reduced version of our method is equivalent to a semismooth Newton algorithm applied to the optimality condition, under a specific choice of the algorithm parameters. We present several computational experiments to show the efficiency of our approach compared to other state-of-the-art algorithms.

show abstract

An inexact successive quadratic approximation method for L-1 regularized optimization

2015

Self Cite

View full text Add to dashboard Cite

We study a Newton-like method for the minimization of an objective function φ that is the sum of a smooth convex function and an ℓ 1 regularization term. This method, which is sometimes referred to in the literature as a proximal Newton method, computes a step by minimizing a piecewise quadratic model q k of the objective function φ. In order to make this approach efficient in practice, it is imperative to perform this inner minimization inexactly. In this paper, we give inexactness conditions that guarantee global convergence and that can be used to control the local rate of convergence of the iteration. Our inexactness conditions are based on a semi-smooth function that represents a (continuous) measure of the optimality conditions of the problem, and that embodies the soft-thresholding iteration. We give careful consideration to the algorithm employed for the inner minimization, and report numerical results on two test sets originating in machine learning.

show abstract

A family of second-order methods for convex $$\ell _1$$-regularized optimization

Cited by 43 publications

References 36 publications

Inexact Orthant-Wise Quasi-Newton Method

Inexact Orthant-Wise Quasi-Newton Method

Second-order orthant-based methods with enriched Hessian information for sparse $$\ell _1$$ ℓ 1 -optimization

An inexact successive quadratic approximation method for L-1 regularized optimization

Contact Info

Product

Resources

About