Variable Importance Analysis in Imbalanced Datasets: A New Approach

Abstract

Decision-making using machine learning requires a deep understanding of the model under analysis. Variable importance analysis provides the tools to assess the importance of input variables when dealing with complex interactions, making the machine learning model more interpretable and computationally more efficient. In classification problems with imbalanced datasets, this task is even more challenging. In this article, we present two variable importance techniques, a nonparametric solution, called mh-χ 2 , and a parametric method based on Global Sensitivity Analysis. The mh-χ 2 employs a multivariate continuous response framework to deal with the multiclass classification problem. Based on the permutation importance framework, the proposed mh-χ 2 algorithm captures the dissimilarities between the distribution of misclassification errors generated by the base learner, Conditional Inference Tree, before and after permuting the values of the input variable under analysis. The GSA solution is based on the Covariance decomposition methodology for multivariate output models. Both solutions will be assessed in a comparative study of several Random Forest-based techniques with emphasis in the multiclass classification problem with different imbalanced scenarios. We apply the proposed techniques in two real application cases in order first, to quantify the importance of the 35 companies listed in the Spanish market index IBEX35 on the economic, political and social uncertainties reflected in economic newspapers in Spain during the first quadrimester of 2020 due to the COVID-19 pandemic and second, to assess the impact of energy factors on the occurrence of spike prices on the Spanish electricity market.

A Machine Learning-Based Surrogate Finite Element Model for Estimating Dynamic Response of Mechanical Systems

“…RF is a nonparametric ML approach for multiclass classification and regression problems [95]. The RF is a wellrecognized and hugely successful bagging type of decision tree ensemble.…”

An optimal variable importance for machine learning classification models using modified simulated annealing algorithm

“…One of the essential outputs of a machine learning algorithm is variable importance (VI) [1]- [2]. This VI indicates the importance of the predictor variables.…”

Analysis of Variable Importance Measurement Techniques for Classification of Road Surfaces

“…As the machine learning accuracy highly dependant upon this stage, for imbalance datasets, accuracy performance becomes a challenging task. Variable importance measurement techniques described in [10], highlights usage of imbalance dataset and shows out performance of proposed method. To achieve high dimensional selection consistency in decision tree algorithm, researchers have presented model selection algorithm named DSTUMP that outperforms in nonlinear additive model settings [10].…”

“…Importance combined with prior knowledge parameters to select features, when applied to soft measuring model, these features have shown increase in performance, as stated in [18]. Similarly permutation based framework, a dissimilarities based algorithm is proposed by [10] researchers that computes variable importance using distribution of misclassification errors. In the area of image classification, researchers have proposed method of quantifying of variable importance, employing concept of game theory and metric of Shapely value which is applicable to any type of model [19].…”