Some sharp performance bounds for least squares regression with L1 regularization

Zhang, Tong

doi:10.1214/08-aos659

Cited by 185 publications

(198 citation statements)

References 19 publications

Supporting

Mentioning

189

Contrasting

Order By: Relevance

“…A variety of sparsity-inducing penalty functions have been proposed to approximate the ℓ 0 term: exponential concave function [4], ℓ p -norm with 0 < p < 1 [15] and p < 0 [71], Smoothly Clipped Absolute Deviation (SCAD) [13], Logarithmic function [82], Capped-ℓ 1 [26] (see (21), (22) and Table 1 in Section 3 for the definition of these functions). Using these approximations, several algorithms have been developed for resulting optimization problems, most of them are in the context of feature selection in classification, sparse regressions or more especially for sparse signal recovery: Successive Linear Approximation (SLA) algorithm [4], DCA (Difference of Convex functions Algorithm) based algorithms [11,12,16,21,28,42,43,51,54,63,65], Local Linear Approximation (LLA) [87], Two-stage ℓ 1 [83], Adaptive Lasso [86], reweighted-ℓ 1 algorithms [8]), reweighted-ℓ 2 algorithms such as Focal Underdetermined System Solver (FOCUSS) ( [18,71,72]), Iteratively reweighted least squares (IRLS) and Local Quadratic Approximation (LQA) algorithm [13,87].…”

Section: Portfolio Selection Problem With Cardinality Constraintmentioning

confidence: 99%

DC approximation approaches for sparse optimization

Thi

Dinh

et al. 2015

European Journal of Operational Research

172

View full text Add to dashboard Cite

Sparse optimization refers to an optimization problem involving the zero-norm in objective or constraints. In this paper, nonconvex approximation approaches for sparse optimization have been studied with a unifying point of view in DC (Difference of Convex functions) programming framework. Considering a common DC approximation of the zero-norm including all standard sparse inducing penalty functions, we studied the consistency between global minimums (resp. local minimums) of approximate and original problems. We showed that, in several cases, some global minimizers (resp. local minimizers) of the approximate problem are also those of the original problem. Using exact penalty techniques in DC programming, we proved stronger results for some particular approximations, namely, the approximate problem, with suitable parameters, is equivalent to the original problem. The efficiency of several sparse inducing penalty functions have been fully analyzed. Four DCA (DC Algorithm) schemes were developed that cover all standard algorithms in nonconvex sparse approximation approaches as special versions. They can be viewed as, an ℓ 1 -perturbed algorithm / reweighted-ℓ 1 algorithm / reweighted-ℓ 1 algorithm. We offer a unifying nonconvex approximation approach, with solid theoretical tools as well as efficient algorithms based on DC programming and DCA, to tackle the zero-norm and sparse optimization. As an application, we implemented our methods for the feature selection in SVM (Support Vector Machine) problem and performed empirical comparative numerical experiments on the proposed algorithms with various approximation functions.

show abstract

Section: Portfolio Selection Problem With Cardinality Constraintmentioning

confidence: 99%

DC approximation approaches for sparse optimization

Thi

Dinh

et al. 2015

European Journal of Operational Research

172

View full text Add to dashboard Cite

show abstract

“…The theoretical properties of the estimator with the Lasso penalty have been extensively studied in the literature. For instance, the sparsity oracle inequalities and statistical rate of convergence of the Lasso estimator are established by Meinshausen and Yu (2009);Bickel et al (2009); Bunea et al (2007); van de Geer (2008); Zhang (2009);Negahban et al (2012), and the variable selection consistency is studied by Meinshausen and Bühlmann (2006); Zhao and Yu (2006); Wainwright (2009). The class of nonconvex penalties includes MCP (Zhang, 2010a), SCAD (Fan and Li, 2001), and capped-L 1 penalty (Zhang, 2010b), among others.…”

Section: Introductionmentioning

confidence: 99%

A general theory of hypothesis tests and confidence regions for sparse high dimensional models

Ning

Liu²

2017

Ann. Statist.

250

327

View full text Add to dashboard Cite

We consider the problem of uncertainty assessment for low dimensional components in high dimensional models. Specifically, we propose a decorrelated score function to handle the impact of high dimensional nuisance parameters. We consider both hypothesis tests and confidence regions for generic penalized M-estimators. Unlike most existing inferential methods which are tailored for individual models, our approach provides a general framework for high dimensional inference and is applicable to a wide range of applications. From the testing perspective, we develop general theorems to characterize the limiting distributions of the decorrelated score test statistic under both null hypothesis and local alternatives. These results provide asymptotic guarantees on the type I errors and local powers of the proposed test. Furthermore, we show that the decorrelated score function can be used to construct point estimators that are semiparametrically efficient and the optimal confidence regions. We also generalize this framework to broaden its applications. First, we extend it to handle high dimensional null hypothesis, where the number of parameters of interest can increase exponentially fast with the sample size. Second, we establish the theory for model misspecification. Third, we go beyond the likelihood framework, by introducing the generalized score test based on general loss functions. Thorough numerical studies are conducted to back up the developed theoretical results.

show abstract

“…We present in Section 7.3 some computation times for the more involved case with non-convex objective function using a block coordinate descent generalized EM algorithm. Regarding statistical theory, almost all results for high-dimensional Lasso-type problems have been developed for convex loss functions, e.g., the squared error in a Gaussian regression (Greenshtein and Ritov, 2004;Meinshausen and Bühlmann, 2006;Zhao and Yu, 2006;Bunea et al, 2007;Zhang and Huang, 2008;Meinshausen and Yu, 2009;Wainwright, 2009;Bickel et al, 2009;Cai et al, 2009b;Candès and Plan, 2009;Zhang, 2009) or the negative log-likelihood in a generalized linear model . We present a non-trivial modification of the mathematical analysis of ℓ 1 -penalized estimation with convex loss to non-convex but smooth likelihood problems.…”

Section: Introductionmentioning

confidence: 99%

ℓ1-penalization for mixture regression models

Städler

Bühlmann

Geer

2010

TEST

220

326

View full text Add to dashboard Cite

We consider a finite mixture of regressions (FMR) model for high-dimensional inhomogeneous data where the number of covariates may be much larger than sample size. We propose an ℓ 1 -penalized maximum likelihood estimator in an appropriate parameterization. This kind of estimation belongs to a class of problems where optimization and theory for non-convex functions is needed. This distinguishes itself very clearly from high-dimensional estimation with convex loss-or objective functions, as for example with the Lasso in linear or generalized linear models. Mixture models represent a prime and important example where non-convexity arises.For FMR models, we develop an efficient EM algorithm for numerical optimization with provable convergence properties. Our penalized estimator is numerically better posed (e.g., boundedness of the criterion function) than unpenalized maximum likelihood estimation, and it allows for effective statistical regularization including variable selection. We also present some asymptotic theory and oracle inequalities: due to non-convexity of the negative log-likelihood function, different mathematical arguments are needed than for problems with convex losses. Finally, we apply the new method to both simulated and real data.

show abstract

Some sharp performance bounds for least squares regression with L1 regularization

Cited by 185 publications

References 19 publications

DC approximation approaches for sparse optimization

DC approximation approaches for sparse optimization

A general theory of hypothesis tests and confidence regions for sparse high dimensional models

ℓ1-penalization for mixture regression models

Contact Info

Product

Resources

About