A New Noisy Random Forest Based Method for Feature Selection

Abstract

Feature selection is an essential pre-processing step in data mining. It aims at identifying the highly predictive feature subset out of a large set of candidate features. Several approaches for feature selection have been proposed in the literature. Random Forests (RF) are among the most used machine learning algorithms not just for their excellent prediction accuracy but also for their ability to select informative variables with their associated variable importance measures. Sometimes RF model over-fits on noisy features, which lead to choosing the noisy features as the informative variables and eliminating the significant ones. Whereas, eliminating and preventing those noisy features first, the low ranked features may become more important. In this study we propose a new variant of RF that provides unbiased variable selection where a noisy feature trick is used to address this problem. First, we add a noisy feature to a dataset. Second, the noisy feature is used as a stopping criterion. If the noisy feature is selected as the best splitting feature, then we stop the creation process because at this level, the model starts to over-fit on the noisy features. Finally, the best subset of features is selected out of the best-ranked feature regarding the Geni impurity of this new variant of RF. To test the validity and the effectiveness of the proposed method, we compare it with RF variable importance measure using eleven benchmarking datasets.

Feature Selection: A Review and Comparative Study

“…Feature selection is an active research filed in machine learning, as it is an important pre-processing, finding success in different real problem applications. In general, feature selection algorithms are categorized into supervised, Semi-supervised and Unsupervised feature selection [2,3,4,5,6]. Supervised feature selection methods usually come in three flavors: Filter, Wrapper and Embedded approach.…”

“…1). As a result, learning models built with the chosen subset of features are more readable and interpretable [3,4,5,25].The main reasons behind using feature selection are: Reducing the storage capacity and execution time, preventing the curse of dimensionality problem, minimizing the over-fitting issue, resulting in improved model generalization and increasing the performance attainability [6].…”

A Robust Feature Construction for Fish Classification Using Grey Wolf Optimizer

“…V e n k a t e s h and A n u r a d h a [19] reviewed several feature selection methods and categorized these methods into three types: filter, wrapper, and embedded methods. Several methods have been used for feature selection, including filter-based [20] and noisy random forest-based method [21].…”

Enhancing the Speed of the Learning Vector Quantization (LVQ) Algorithm by Adding Partial Distance Computation

“…Machine Learning (ML) is considered as one important way for making benefit from this data. Moreover, the rapid technological advancements and large data production need to upgrade or change conventional techniques [5]. ML is the study that aims to use the computer algorithm that developed its behaviour over time based on the experience.…”