Filter Methods for Feature Selection in Supervised Machine Learning Applications -- Review and Benchmark

Abstract

The amount of data for machine learning (ML) applications is constantly growing. Not only the number of observations, especially the number of measured variables (features) increases with ongoing digitization. Selecting the most appropriate features for predictive modeling is an important lever for the success of ML applications in business and research. Feature selection methods (FSM) that are independent of a certain ML algorithm-so-called filter methods-have been numerously suggested, but little guidance for researchers and quantitative modelers exists to choose appropriate approaches for typical ML problems. This review synthesizes the substantial literature on feature selection benchmarking and evaluates the performance of 58 methods in the widely used R environment. For concrete guidance, we consider four typical dataset scenarios that are challenging for ML models (noisy, redundant, imbalanced data and cases with more features than observations). Drawing on the experience of earlier benchmarks, which have considered much fewer FSMs, we compare the performance of the methods according to four criteria (predictive performance, number of relevant features selected, stability of the feature sets and runtime). We found methods relying on the random forest approach, the double input symmetrical relevance filter (DISR) and the joint impurity filter (JIM) were well-performing candidate methods for the given dataset scenarios.

Effectiveness of Variable Selection Methods for Machine Learning and Classical Statistical Models

“…The methods can be divided into methods related to the data model so-called wrapper methods (e.g. recursive feature elimination, heuristic methods) or embedded methods (Random Forests, LASSO, Ridge Regression) [Lal et al 2006] and model-independent filtering methods [Sánchez-Maroño et al 2007;Hopf 2021] e.g. based on variable correlation or mutual information (MI) measures [Vergara 2014;Gajowniczek et al 2022].…”

Selekcja zmiennych metodami statystycznymi i uczenia maszynowego. Porównanie podejść na przykładzie danych finansowych

“…2006] oraz niezależne od modelu metody filtrowania [Sánchez-Maroño i in. 2007;Hopf 2021] np. oparte na korelacji zmiennych czy mierze informacji wzajemnej (MI) [Vergara 2014;Gajowniczek i in.…”

On Machine Learning Techniques Toward Path Loss Modeling in 5g and Beyond Wireless Systems

“…Regarding predictor selection techniques in supervised algorithms, filter, and wrapper methods are usually employed to forecast the most relevant predictors for good performance and to understand the model's response [56]. Filter methods assess the importance of the predictor based on the dataset, independently of the applied ML algorithm and its performance [57], for example, using mutual information (MI) [58]. The MI is calculated on the dataset to provide pre-interpretations, i.e., before training, aiming to measure the correlation between predictors and the output.…”