Automatic Feature Selection via Weighted Kernels and Regularization

Abstract

Selecting important features in nonlinear kernel spaces is a difficult challenge in both classification and regression problems. This article proposes to achieve feature selection by optimizing a simple criterion: a feature-regularized loss function. Features within the kernel are weighted, and a lasso penalty is placed on these weights to encourage sparsity. This feature-regularized loss function is minimized by estimating the weights in conjunction with the coefficients of the original classification or regression problem, thereby automatically procuring a subset of important features. The algorithm, KerNel Iterative Feature Extraction (KNIFE), is applicable to a wide variety of kernels and high-dimensional kernel problems. In addition, a modification of KNIFE gives a computationally attractive method for graphically depicting nonlinear relationships between features by estimating their feature weights over a range of regularization parameters. The utility of KNIFE in selecting features through simulations and examples for both kernel regression and support vector machines is demonstrated. Feature path realizations also give graphical representations of important features and the nonlinear relationships among variables. Supplementary materials with computer code and an appendix on convergence analysis are available online.

Variable Selection in Kernel Regression Using Measurement Error Selection Likelihoods

“…SPAM (sparse additive models) is similar to COSSO in that it truncates complexity, but it allows p ≫ n (Ravikumar et al, 2009). Kernel iterative feature extraction (KNIFE) by Allen (2013) imposes L 1 -regularization on L 2 -penalized splines.…”

“…We use the default GCV criterion for MARS. For KNIFE, we fix λ 1 = 1 and use a radial kernel with γ = 1/ p as suggested in Allen (2013). The weight power for the ACOSSO is fixed at γ = 2, as suggested by Storlie et al (2011).…”

Probabilistic Modeling of Imaging, Genetics and Diagnosis

“…Note that this is not the case in the classical kernel-based approaches where the prediction is the only goal but not features selection. Previously demonstrated methods for feature selection using kernel machines [52] lack a probabilistic model required by our approach. Further extension of our model to those cases is possible but beyond the scope of this paper.…”

Kernel continuum regression

“…To achieve that goal, one may consider more flexible forms of kernel functions to allow the method to automatically remove variables. Existing literature in nonlinear variable selection, for example, the COSSO (Lin and Zhang, 2007) and KNIFE (Allen, 2013), can be useful here. In that case, the corresponding computational algorithm can be much more challenging.…”