Induction of Decision Trees via Evolutionary Programming

Abstract

Decision trees have been used extensively in cheminformatics for modeling various biochemical endpoints including receptor-ligand binding, ADME properties, environmental impact, and toxicity. The traditional approach to inducing decision trees based upon a given training set of data involves recursive partitioning which selects partitioning variables and their values in a greedy manner to optimize a given measure of purity. This methodology has numerous benefits including classifier interpretability and the capability of modeling nonlinear relationships. The greedy nature of induction, however, may fail to elucidate underlying relationships between the data and endpoints. Using evolutionary programming, decision trees are induced which are significantly more accurate than trees induced by recursive partitioning. Furthermore, when assessed on previously unseen data in a 10-fold cross-validated manner, evolutionary programming induced trees exhibit a significantly higher accuracy on previously unseen data. This methodology is compared to single-tree and multiple-tree recursive partitioning in two domains (aerobic biodegradability and hepatotoxicity) and shown to produce less complex classifiers with average increases in predictive accuracy of 5-10% over the traditional method.

A Survey of Evolutionary Algorithms for Decision-Tree Induction

“…Some approaches propose additional constraints during initialization in order to to guarantee the logic validity of the created decision trees. For instance, DeLisle and Dixon [51] state that their method initializes decision trees "at random based upon the training set and given mild constraints on the minimum number of observations allowed in terminal or leaf nodes".…”

“…DeLisle et al [51] propose the evaluation of accuracy and complexity of decision trees by using minimum description length (MDL) as fitness function, defined as follows:…”

“…In evolutionary induction of decision trees, the most frequently used approach for selection is tournament selection.This selection method is used in [28], [38], [45], [48], [51], [52], [56]- [59], [64], [66], [67], [75], [76], [85].…”

“…Another popular choice in EAs for decision tree induction is the roulette wheel selection.Works that implement this strategy for evolving decision trees are [35]- [37], [42]- [44], [48], [51], [55], [60]- [63], [73], [82], [86], [87].…”

Rule Induction for Classification Using Multi-objective Genetic Programming

“…While there is much literature on the use of genetic programming to optimize trees for the purposes of classification, this mostly concentrates on the optimization of decision trees, e.g. [10,11,12,13]. In particular, Mugambi and Hunter [14] apply multi-objective genetic programming to decision tree induction, optimizing both tree accuracy and tree simplicity.…”