Feature Selection Based on Divergence Functions: A Comparative Classiffication Study

Abstract

Due to the extensive use of high-dimensional data and its application in a wide range of scientifc felds of research, dimensionality reduction has become a major part of the preprocessing step in machine learning. Feature selection is one procedure for reducing dimensionality. In this process, instead of using the whole set of features, a subset is selected to be used in the learning model. Feature selection (FS) methods are divided into three main categories: flters, wrappers, and embedded approaches. Filter methods only depend on the characteristics of the data, and do not rely on the learning model at hand. Divergence functions as measures of evaluating the differences between probability distribution functions can be used as flter methods of feature selection. In this paper, the performances of a few divergence functions such as Jensen-Shannon (JS) divergence and Exponential divergence (EXP) are compared with those of some of the most-known flter feature selection methods such as Information Gain (IG) and Chi-Squared (CHI). This comparison was made through accuracy rate and F1-score of classifcation models after implementing these feature selection methods.

Application of Machine Learning Techniques to Help in the Feature Selection Related to Hospital Readmissions of Suicidal Behavior

“…The chi-square analysis is a method that is widely used to identify attributes that have the greatest bearing on accounting for a response variable subject to study. In Pourmand et al ( 2021 ), this method is explained as being used to select attributes based on the statistical distribution of data, although, in many applications, the probability distribution function is unknown and difficult to estimate (Berenfeld & Hoffmann, 2021 ). That is why a decision was made to use a tree-based iterative method—the CHAID analysis (Jojoa et al, 2021 )—complemented by a machine learning-based attributed selection, as the idea was to improve generalisation of the results obtained for these types of clinical issues in which attribute selection is of great importance.…”