Combination of Ensembles of Regularized Regression Models with Resampling-Based Lasso Feature Selection in High Dimensional Data

Abstract

In high-dimensional data, the performances of various classifiers are largely dependent on the selection of important features. Most of the individual classifiers with the existing feature selection (FS) methods do not perform well for highly correlated data. Obtaining important features using the FS method and selecting the best performing classifier is a challenging task in high throughput data. In this article, we propose a combination of resampling-based least absolute shrinkage and selection operator (LASSO) feature selection (RLFS) and ensembles of regularized regression (ERRM) capable of dealing data with the high correlation structures. The ERRM boosts the prediction accuracy with the top-ranked features obtained from RLFS. The RLFS utilizes the lasso penalty with sure independence screening (SIS) condition to select the top k ranked features. The ERRM includes five individual penalty based classifiers: LASSO, adaptive LASSO (ALASSO), elastic net (ENET), smoothly clipped absolute deviations (SCAD), and minimax concave penalty (MCP). It was built on the idea of bagging and rank aggregation. Upon performing simulation studies and applying to smokers’ cancer gene expression data, we demonstrated that the proposed combination of ERRM with RLFS achieved superior performance of accuracy and geometric mean.

Hybrid Rank Aggregation (HRA): A novel rank aggregation method for ensemble-based feature selection

“…The feature performance FP of all features for m models is obtained to create a feature performance matrix FP mat . The different metrics are used as FP in literature like a coefficient estimate of feature [15,16], goodness of fit [18], feature importance [28] and feature significance [21]. For illustration purposes, feature importance a is used as FP .…”

“…The different metrics are used as ‫ܲܨ‬ in literature like a coefficient estimate of feature [15,16], goodness of fit [18], feature importance [28] and feature significance [21]. For illustration purposes, feature importance ߙ is used as ‫.ܲܨ‬ Since, in any model, only ݈| ݈ ‫‬ features are used, thus many cells in the ‫ܲܨ‬ ௧ will not have a ߙ value.…”

“…EFS is a multi-step approach where literature provides different techniques for each step (Figure 1). The first step creates multiple datasets from the original data either by sampling of features [15], samples [16,17] or both [18,19], or by duplicating the original dataset [15,20,21]. Each of these datasets can be processed incorporating interaction terms for the features [19] and undergoing feature transformation [18].…”

“…The approach which uses the original dataset and multiple base techniques to build the model is called the heterogenous EFS approach [20]. The third step is the extraction of feature performance from each of the models using either single [16] or multiple metrics [17]. The feature performance is pooled or aggregated in the next step and is referred to as rank aggregation (RA).…”

Hyperspectral band selection and modeling of soil organic matter content in a forest using the Ranger algorithm

“…These two algorithms are suitable for the resampling and feature selection of high-dimensional data with many features [30,31]. Moreover, these two algorithms have been applied well in disease diagnosis and image processing [32,33]. One of the goals of this study was to determine whether ML techniques can improve the selection of spectral bands.…”