An inexact successive quadratic approximation method for L-1 regularized optimization

Abstract

We study a Newton-like method for the minimization of an objective function φ that is the sum of a smooth convex function and an ℓ 1 regularization term. This method, which is sometimes referred to in the literature as a proximal Newton method, computes a step by minimizing a piecewise quadratic model q k of the objective function φ. In order to make this approach efficient in practice, it is imperative to perform this inner minimization inexactly. In this paper, we give inexactness conditions that guarantee global convergence and that can be used to control the local rate of convergence of the iteration. Our inexactness conditions are based on a semi-smooth function that represents a (continuous) measure of the optimality conditions of the problem, and that embodies the soft-thresholding iteration. We give careful consideration to the algorithm employed for the inner minimization, and report numerical results on two test sets originating in machine learning.

Global convergence rate analysis of unconstrained optimization methods based on probabilistic models

“…In the nonlinear optimization setting, the complexity of various unconstrained methods has been derived under exact derivative information [7,8,17], and also under inexact information, where the errors are bounded in a deterministic fashion [3,6,11,14,20]. In all the cases of the deterministic inexact setting, traditional optimization algorithms such as line search, trust region or adaptive regularization algorithms are applied with little modification and work in practice as well as in theory, while the error is assumed to be bounded in some decaying manner at each iteration.…”

Inexact Orthant-Wise Quasi-Newton Method

“…Byrd, Nocedal and Oztoprak [5] proposed an Inexact Newton-like Method, we also call it proximal Newton Method. This method compute an inexact solution of the piecewise quadratic model satisfying some inexactness conditions at every iteration, consider the following optimization problem: …”

“…Lee, Sun and Saunders [28] presented an inexact proximal Newton method to solve problem(1) and establish several local convergence results. Byrd, Nocedal and Oztoprak [5] proposed an inexact successive quadratic approximation method(SQA), this method also use a quadratic model to approximate the objective function. Instead of compute the exact solution to the quadratic model, SQA compute an approximate solution satisfying some inexactness condition at every iteration.…”

A second-order method for convex1-regularized optimization with active-set prediction

“…These include first-order methods, such as ISTA, *Corresponding author. Email: nitishkeskar2012@u.northwestern.edu SpaRSA and FISTA [3,10,37], and proximal Newton methods that compute a step by minimizing a piecewise quadratic model of (1) using (for example) a coordinate descent iteration [5,16,19,23,26,29,32,38]. The proposed algorithm also differs from methods that solve (1) by reformulating it as a bound constrained problem [12,18,27,28,33,35], and from recent methods that are specifically designed for the case when f is a convex quadratic [9,30,34].…”

“…Implementations of the proximal Newton method employ adaptive techniques (heuristics or rules based in randomized analysis) [16,26,38], or rules based on an optimality measure [5,19]. Our implementation makes use of the classic termination criteria based on the relative error in the residue of the linear system [22].…”