Variable Selection Using Adaptive Nonlinear Interaction Structures in High Dimensions

Abstract

Numerous penalization based methods have been proposed for fitting a traditional linear regression model in which the number of predictors, p, is large relative to the number of observations, n. Most of these approaches assume sparsity in the underlying coefficients and perform some form of variable selection. Recently, some of this work has been extended to non-linear additive regression models. However, in many contexts one wishes to allow for the possibility of interactions among the predictors. This poses serious statistical and computational difficulties when p is large, as the number of candidate interaction terms is of order p 2 . We introduce a new approach, "Variable selection using Adaptive Non-linear Interaction Structures in High dimensions" ( VAN-ISH), that is based on a penalized least squares criterion and is designed for high dimensional non-linear problems. Our criterion is convex and enforces the heredity constraint, in other words if an interaction term is added to the model, then the corresponding main effects are automatically included. We provide theoretical conditions under which VANISH will select the correct main effects and interactions. These conditions suggest that VANISH should outperform certain natural competitors when the true interaction structure is sufficiently sparse. Detailed simulation results are also provided, demonstrating that VAN-ISH is computationally efficient and can be applied to non-linear models involving thousands of terms while producing superior predictive performance over other approaches.

Sparse Regularization for Bi-Level Variable Selection

“…The CAP can also be seen as a part of the composite penalties (5), so we added the CAP to Table 1. Choi et al (2010), Radchenko and James (2010), Bach et al (2012) and Bien et al (2013) also considered methods for hierarchical selection. This paper focuses on the setting where there are grouped structures without any overlap.…”

Bias-corrected inference for multivariate nonparametric regression: Model selection and oracle property

“…) and h (j) is the vector of the bandwidths except h j , with a(x, K , h (j) ) and b j (x, K ) defined in (13). This assumption has a similar role as the assumption (4) in [19].…”

“…1(a). Now consider the bias functional in (13). As a function of h j , its behavior is depicted in Fig.…”

“…The derivations are more difficult, above all, in order to derive the bias of our estimators. In literature, there are papers as [13,14], which can allow correlated covariates for nonparametric models in order to make variable selection. However, they use an additive way to deal with the general nonparametric unknown function and the methods are based on penalized approach with regularization parameters to estimate.…”

An efficient algorithm for weak hierarchical lasso

“…However, the all-pairs Lasso estimator does not account for any structural information which has been shown to be important for prediction and interpretation of the high dimensional interaction regression model [2,30,25,29,6]. In statistics, a hierarchical structure between main effects and interaction effects has been shown to be very effective in constraining the search space and identifying important individual features and interactions [2,30,25,29,6]. Specifically, the hierarchical constraint requires that an interaction term xixj is selected in the model only if the main effects xi and/or xj are included.…”