High-dimensional variable selection

Abstract

This paper explores the following question: what kind of statistical guarantees can be given when doing variable selection in high dimensional models? In particular, we look at the error rates and power of some multi-stage regression methods. In the first stage we fit a set of candidate models. In the second stage we select one model by cross-validation. In the third stage we use hypothesis testing to eliminate some variables. We refer to the first two stages as “screening” and the last stage as “cleaning.” We consider three screening methods: the lasso, marginal regression, and forward stepwise regression. Our method gives consistent variable selection under certain conditions.

Estimation and Inference of Heterogeneous Treatment Effects using Random Forests

“…consistency results; here, we use honesty following, e.g., Wasserman and Roeder [2009]. We note that there have been some recent theoretical investigations of non-honest forests, including Scornet et al [2015] and Wager and Walther [2015].…”

Independent filtering increases detection power for high-throughput experiments

“…Conceptually similar screening approaches have also been proposed for variable selection in high-dimensional regression models (13,14). In filtering for microarray applications, the data are first used to identify and remove a set of genes which seem to generate uninformative signal.…”

Direct effects testing: A two‐stage procedure to test for effect size and variable importance for correlated binary predictors and a binary response

“…A lasso 'false find' (ffind) occurs when a non-zero coefficient is assigned to a noncausal predictor. We additionally compare with the 'screen and clean' (S&C) method of Wasserman and Roeder [2009], where the strength of the penalty in the 'screen' stage is chosen via BIC. In Wasserman and Roeder [2009] cross validation is used to determine the penalty for the 'screen' stage -this leads to more variables being carried forward to the 'clean' stage, compared with BIC, and hence more true and false finds.…”

“…We additionally compare with the 'screen and clean' (S&C) method of Wasserman and Roeder [2009], where the strength of the penalty in the 'screen' stage is chosen via BIC. In Wasserman and Roeder [2009] cross validation is used to determine the penalty for the 'screen' stage -this leads to more variables being carried forward to the 'clean' stage, compared with BIC, and hence more true and false finds. Due to the high multicollinearity in this particular dataset, the increase in false finds was particularly damaging for both the lasso and the 'screen and clean' methods, so using BIC seemed to give more favourable results for these methods.…”

“…In this paper we are interested in recovering the so-called 'true sparsity pattern' [Wasserman and Roeder, 2009], in which we search for a subset of predictors deemed to have a 'direct effect' on the response -that is an effect that is causally attributed to the predictor in question rather than being due to the correlation of the predictor with other important predictors. We wish to find a sparse solution to the regression given in Equation (1) and in particular carry out significance tests of variable importance.…”