An Information-Theoretic Approach for Setting the Optimal Number of Decision Trees in Random Forests

Abstract

Data Classification is a process within the Data Mining and Machine Learning field which aims at annotating all instances of a dataset by so-called class labels. This involves in creating a model from a training set of data instances which are already labeled, possibly being this model also used to define the class of data instances which are not classified already. A successful way of performing the classification process is provided by the algorithm Random Forests (RF), which is itself a type of Ensemble-based Classifier. An ensemble-based classifier increases the accuracy of the class label assigned to a data instance by using a set of classifiers that are modeled on different, but possibly overlapping, instance sets, and then combining the so-obtained intermediate classification results. To this end, RF particularly makes use of a number of decision trees to classify an instance, then taking the majority of votes from these trees as the final classifier. The latter one is a critical task of algorithm RF, which heavily impacts on the accuracy of the final classifier. In this paper, we propose a variation of algorithm RF, namely adjusting one of the two parameters that RF takes, the number of decision trees, dependant on a meaningful relation between the dataset predictive power rating and the number of trees itself, with the goal of improving accuracy and performance of the algorithm. This is finally demonstrated by our comprehensive experimental evaluation on several clean datasets.

Dynamic Nonparametric Random Forest Using Covariance

“…To optimize the number of trees while keeping the classification accuracy close to or higher than that of the original RF algorithm, Cuzzocrea et al proposed a new algorithm 7end while (8) end procedure Algorithm 1: Overall Operation. [18]. Based on the relationship between the predictive power which means the percentage of positively classified cases of instances of that dataset and the number of trees in a forest, they proposed how to optimize the number of trees in RF using an information-theoretic approach.…”

“…for all ∈ do (4) Count either , or from the return (5) value of Test4Classification( , ); (6) end for (7) = /( + ); (8) = /( + ); (9) = F Measure( , ); (10) ← ; (11) else (12) for all ∈ do (13) for all 1 ≤ ≤ do (14) Count either , or from the return (15) value of Test4Classification( , ); (16) end for (17) end for (18) = /( + ); (19) = node based on the best attribute is included into (Line (6)). Based on the best attribute, C4.5 splits and, thus, generates .…”

“…To find the optimal value of , Cuzzocrea et al proposed an algorithm that gradually increased the number of trees [18] and found the best classifiers than the others. Even though Cuzzocrea et al 's algorithm minimizes the number of trees composing the forest, it requires much learning time and much memory usage.…”

“…While keeping the best classification accuracy close to that of the original RF algorithm [12], the proposed algorithm reduces the number of trees composing the forest. Compared to Cuzzocrea et al 's algorithm [18], the proposed algorithm does not need to know the predefined number of trees and a certain a priori information. Also, compared to P. Latinne et al 's algorithm [19], the proposed algorithm uses the covariance between the number of trees and -measure for a forest at each iteration.…”

Resampling Imbalanced Class and the Effectiveness of Feature Selection Methods for Heart Failure Dataset

“…Accuracy is simply measured by the possibility that the algorithm can predict negative and positive instances correctly. 34,35 as:…”