Sure Independence Screening for Ultrahigh Dimensional Feature Space

Abstract

DiscussionWe congratulate the authors for their thought-provoking and fascinating work on a fundamental yet challenging topic in variable selection. Driven by the pressing need of high dimensional data analysis in many fields, the problem of dimension reduction without losing relevant information becomes increasingly important. Fan and Lv successfully tackled the extremely challenging case, where log(p) = O(n ξ ), ξ > 0. The proposed Sure Independence Screening (SIS) is a state of the art method for high dimensional variable screening: simple, powerful, and having optimal properties. This work is a substantial contribution to the area of variable selection and will also make a significant impact in other scientific fields.. Extension to nonparametric modelsIn linear models, marginal correlation coefficients between linear predictors and the response are effective measures to capture strength of their linear relationship. However, correlation coefficients generally do not work for ranking nonlinear effects. Consider the additive model, where f j takes an arbitrary nonlinear function form. Motivated by the ranking idea of the SIS, one could first fit a univariate smoother for each predictor and then use some marginal statistics to rank the covariates. Many interesting questions arise in this approach. Firstly, what are good measures to characterize the strength of the nonlinear relationship fully? Possible choices include nonparametric test statistics, p-values, and goodness-of-fit statistics like R 2 . But which is best? Also, how do we develop the consistent selection theory for the procedure of screening nonlinear effects? All of these questions are challenging because of the complicated estimation that is involved in nonparametric modeling. It would be interesting to explore whether and how the SIS can be extended to this context. Connection to multiple hypotheses testing and false discovery rate controlThe variable selection problem can be regarded as the problem of testing multiple hypotheses: H 1 : β 1 = 0, …, H p : β p = 0. Screening important variables is hence equivalent to identifying the hypotheses to be rejected. The false discovery rate (Benjamini and Hochberg, 1995) has been developed to control the proportion of false rejections. Some consistent procedures based on individual tests of each parameter have been developed (Potscher 1983;Bauer et al. 1988). Recently, Bunea et al., (2006) considered the case when p increases with n, and showed the false discover rate or Bernoulli adjustment can lead to consistent selection of variables under certain conditions. Their method is based on the ordered p-values of individual t-statistics NIH Public Access

Variable selection for sparse high-dimensional nonlinear regression models by combining nonnegative garrote and sure independence screening

“…Moreover, the nonnegative garrote cannot be applied when the sample size is smaller than the number of covariates. To overcome the challenge of high dimensionality, we consider the idea of sure independence screening for linear or generalized linear regression models proposed by Fan and Lv (2008) and Fan and Song (2010), and extend it to the nonlinear additive regression models.…”

“…In Fan and Lv (2008) and Fan and Song (2010), the covariates are standardized, i.e. $\mathrm{normalE}{X}_j^2=1$, j = 1, ···, p n , so the magnitude of the coefficient estimate of the marginal model can preserve the nonsparsity of the joint model.…”

“…Following the variable screening, we can apply a more refined model selection technique to choose important covariates in the nonlinear additive model conditional on the selected variable set | |, and for this purpose, we use the nonnegative garrote for nonlinear additive models described in Section 2. Based on the philosophy in Fan and Lv (2008), independence screening should be applied when p n is very large; while for moderately high p n , one should use penalized methods for variable selection. In our case, the penalized model selection via the nonnegative garrote for nonlinear additive models requires to obtain the initial parameter estimates through the NLS optimization and it is very difficult to get good initial estimates when p n is large.…”

“…However, as the number of covariates p grows, it becomes quite difficult to get good estimates for α j . To overcome the challenge of high dimensionality, we adopt the idea of independence screening introduced by Fan and Lv (2008). The sure independence screening (SIS) method performs efficient dimension reduction via marginal correlation learning for ultra-high dimensional feature selection.…”

“…This nonlinear independence screening procedure can significantly reduce the dimension of the covariates, and more importantly, its sure screening property ensures that all the important covariates will be retained with probability tending to 1. Our method is not a trivial extension of Fan and Lv (2008) and Fan and Song (2010). For the nonlinear model (2), the minimum distinguishable signal of the marginal screening is closely related to the stochastic and numerical errors in the optimization of the nonlinear parameters.…”