Evaluating the Impact of GINI Index and Information Gain on Classification using Decision Tree Classifier Algorithm*

Abstract

Decision tree is a supervised machine learning algorithm suitable for solving classification and regression problems. Decision trees are recursively built by applying split conditions at each node that divides the training records into subsets with output variable of same class. The process starts from the root node of the decision tree and progresses by applying split conditions at each non-leaf node resulting into homogenous subsets. However, achieving pure homogenous subsets is not possible. Therefore, the goal at each node is to identify an attribute and a split condition on that attribute that minimizes the mixing of class labels, thus resulting into nearly pure subsets. Several splitting indices were proposed to evaluate the goodness of the split, common ones being GINI index and Information gain. The aim of this study is to conduct an empirical comparison of GINI index and information gain. Classification models are built using decision tree classifier algorithm by applying GINI index and Information gain individually. The classification accuracy of the models is estimated using different metrics such as Confusion matrix, Overall accuracy, Per-class accuracy, Recall and Precision. The results of the study show that, regardless of whether the dataset is balanced or imbalanced, the classification models built by applying the two different splitting indices GINI index and information gain give same accuracy. In other words, choice of splitting indices has no impact on performance of the decision tree classifier algorithm.

How to Effectively Collect and Process Network Data for Intrusion Detection?

“…Another way that shows the importance of a feature is to rank the tree due to the decrease in impurity (Gini impurity) relative to all trees [56]. According to the principle of the algorithm, the most impure trees are at the beginning and the least impure trees are at the end.…”

Pendugaan Fraud Detection pada kartu kredit dengan Machine Learning

“…Gini impurity adalah ukuran seberapa sering elemen yang dipilih secara acak dari himpunan akan diberi label yang salah jika diberi label secara acak sesuai dengan distribusi label di dalam subset. Penggunaan indeks gini dan perolehan informasi sebelum maupun sesudah data set balanced tidak mempengaruhi performa model [9].…”

An Unsupervised Model for Identifying and Characterizing Dark Web Forums

“…There are different selection measures for the splitting criteria of the decision tree that performs the data portioning into best possible manner. Information Gain, Gain Ratio, and Gini Index are most popular examples (Tangirala, 2020). We have measured our evaluation with Information Gain and Gini Index.…”