Calibrating nonconvex penalized regression in ultra-high dimension

Wang, Lan; Kim, Yongdai; Li, Runze

doi:10.1214/13-aos1159

Cited by 149 publications

(150 citation statements)

References 37 publications

Supporting

Mentioning

149

Contrasting

Order By: Relevance

“…More recently, Wang et al (2013) proposed a two-step approach named calibrated CCCP which achieve strong oracle properties when using the Lasso estimator as initialization. Our work differs from theirs in two aspects.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

I-LAMM for sparse learning: Simultaneous control of algorithmic complexity and statistical error

et al. 2018

View full text Add to dashboard Cite

We propose a computational framework named iterative local adaptive majorize-minimization (I-LAMM) to simultaneously control algorithmic complexity and statistical error when fitting high dimensional models. I-LAMM is a two-stage algorithmic implementation of the local linear approximation to a family of folded concave penalized quasi-likelihood. The first stage solves a convex program with a crude precision tolerance to obtain a coarse initial estimator, which is further refined in the second stage by iteratively solving a sequence of convex programs with smaller precision tolerances. Theoretically, we establish a phase transition: the first stage has a sublinear iteration complexity, while the second stage achieves an improved linear rate of convergence. Though this framework is completely algorithmic, it provides solutions with optimal statistical performances and controlled algorithmic complexity for a large family of nonconvex optimization problems. The iteration effects on statistical errors are clearly demonstrated via a contraction property. Our theory relies on a localized version of the sparse/restricted eigenvalue condition, which allows us to analyze a large family of loss and penalty functions and provide optimality guarantees under very weak assumptions (For example, I-LAMM requires much weaker minimal signal strength than other procedures). Thorough numerical results are provided to support the obtained theory.

show abstract

Section: Conclusion and Discussionmentioning

confidence: 99%

I-LAMM for sparse learning: Simultaneous control of algorithmic complexity and statistical error

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Recently, Fan et al [9] have shown that the folded concave penalization methods enjoy the strong oracle property for high-dimensional sparse estimation. Important insight has also been gained through the recent work on theoretical analysis of the global solution [19][20][21].…”

Section: Journal Of Probability and Statisticsmentioning

confidence: 99%

“…Recently, Dicker et al [6] proposed a BIC-like tuning parameter selector. Wang et al [21] extended the work of [24,25] for BIC on highdimensional least squares regression; they proposed a highdimensional BIC for a nonconvex penalized solution path.…”

Section: Journal Of Probability and Statisticsmentioning

confidence: 99%

Variable Selection and Parameter Estimation with the Atan Regularization Method

Wang

Zhu

2016

Journal of Probability and Statistics

View full text Add to dashboard Cite

Variable selection is fundamental to high-dimensional statistical modeling. Many variable selection techniques may be implemented by penalized least squares using various penalty functions. In this paper, an arctangent type penalty which very closely resembles 0 penalty is proposed; we call it Atan penalty. The Atan-penalized least squares procedure is shown to consistently select the correct model and is asymptotically normal, provided the number of variables grows slower than the number of observations. The Atan procedure is efficiently implemented using an iteratively reweighted Lasso algorithm. Simulation results and data example show that the Atan procedure with BIC-type criterion performs very well in a variety of settings.

show abstract

“…The class of nonconvex penalties includes MCP (Zhang, 2010a), SCAD (Fan and Li, 2001), and capped-L 1 penalty (Zhang, 2010b), among others. The estimation consistency and oracle properties of the nonconvex estimators are investigated by Fan et al (2012); Fan and Lv (2011); Loh and Wainwright (2013); Wang et al (2013a); Zhang et al (2013); Wang et al (2013b). Though significant progress has been made towards understanding the estimation theory for high dimensional models, how to quantify the uncertainty of the obtained penalized estimators remains largely unexplored.…”

Section: Introductionmentioning

confidence: 99%

A general theory of hypothesis tests and confidence regions for sparse high dimensional models

Ning

Liu²

2017

Ann. Statist.

249

327

View full text Add to dashboard Cite

We consider the problem of uncertainty assessment for low dimensional components in high dimensional models. Specifically, we propose a decorrelated score function to handle the impact of high dimensional nuisance parameters. We consider both hypothesis tests and confidence regions for generic penalized M-estimators. Unlike most existing inferential methods which are tailored for individual models, our approach provides a general framework for high dimensional inference and is applicable to a wide range of applications. From the testing perspective, we develop general theorems to characterize the limiting distributions of the decorrelated score test statistic under both null hypothesis and local alternatives. These results provide asymptotic guarantees on the type I errors and local powers of the proposed test. Furthermore, we show that the decorrelated score function can be used to construct point estimators that are semiparametrically efficient and the optimal confidence regions. We also generalize this framework to broaden its applications. First, we extend it to handle high dimensional null hypothesis, where the number of parameters of interest can increase exponentially fast with the sample size. Second, we establish the theory for model misspecification. Third, we go beyond the likelihood framework, by introducing the generalized score test based on general loss functions. Thorough numerical studies are conducted to back up the developed theoretical results.

show abstract

Calibrating nonconvex penalized regression in ultra-high dimension

Cited by 149 publications

References 37 publications

I-LAMM for sparse learning: Simultaneous control of algorithmic complexity and statistical error

I-LAMM for sparse learning: Simultaneous control of algorithmic complexity and statistical error

Variable Selection and Parameter Estimation with the Atan Regularization Method

A general theory of hypothesis tests and confidence regions for sparse high dimensional models

Contact Info

Product

Resources

About