The stability of different aggregation techniques in ensemble feature selection

Abstract

To mitigate the curse of dimensionality in high-dimensional datasets, feature selection has become a crucial step in most data mining applications. However, no feature selection method consistently delivers the best performance across different domains. For this reason and in order to improve the stability of the feature selection process, ensemble feature selection frameworks have become increasingly popular. While many have examined the construction of ensemble techniques under various considerations, little work has been done to shed light on the influence of the aggregation process on the stability of the ensemble feature selection. In contribution to this field, this work aims to explore the impact of some selected aggregation strategies on the ensemble’s stability and accuracy. Using twelve classification real datasets from various domains, the stability and accuracy of five different aggregation techniques were examined under four standard filter feature selection methods. The experimental analysis revealed significant differences in both the stability and accuracy behavior of the ensemble under different aggregations, especially between score-based and rank-based aggregation strategies. Moreover, it was observed that the simpler score-based strategies based on the Arithmetic Mean or L2-norm aggregation appear to be efficient and compelling in most cases. Given the data structure or associated application domain, this work’s findings can guide the construction of feature selection ensembles using the most efficient and suitable aggregation rules.

Profile Verification and Secured Social Engineering with Machine Learning Models

“…Calculation of absolute error has been described in equation (15). Absolute Error = Absolute (Predictions -Test Labels) (15) Next, the aim is to find the Mean Absolute Percentage Error as per equation ( 16) which aids in the calculation of the Accuracy as per (17)…”

“…To understand the different divisions of Aggregations after Bootstrapping, it is important to determine the method of processing. As per Reem Salman et al [17], to determine stability one needs to focus on feature selection, followed by rank-based and score-based aggregation. A few methods for Rank based aggregation are Borda's, Stuart, ROBUST Rank Aggregation, SVM Rank, and for Score Based aggregation namely, Borda's methods of Arithmetic Mean, Median, Geometric Mean, and L2 Norm.…”

“…The first one highlights the need for normalization. The need to normalize the data lies in stability as it forms a common playing ground for the dataset by reducing the variations within the data points [17]. As mentioned previously, algorithms such as LSTM work highly on correlation, to reduce bias in a limited dataset, it is important to introduce factors that decrease the need for correlation.…”

Feature Selection with Integrated Gaussian Seahorse Optimization Data Mining for Cross-border Business Cooperation between the Malaysian Medical Industry and Tourism Industry

“…In [16] evaluated on crop prediction based on soil and environmental characteristics using feature selection techniques. It explores the application of feature selection algorithms to identify the most relevant soil and environmental variables for predicting crop yield or other crop-related parameters.…”

Enhancing Olive Phenology Prediction: Leveraging Market Basket Analysis and Weighted Metrics for Optimal Feature Group Selection

“…Selection techniques can be divided according to the contribution of their application. Some cover the dimensionality of the original data set with Feature Extraction techniques like Principal Component Analysis [5], t-Distributed Stochastic Neighbor Embedding [6], or aggregation models [7]. Others focus on creating a feature ranking [8] to select the most influential features, through methods such as Feature Importance from Tree-Based Models [9], Recursive Feature Elimination (RFE) [10], Mutual Information [11], Correlation-Based Feature Ranking [12], between others.…”