Evolving decision trees using oracle guides

Abstract

Abstract-Some data mining problems require predictive models to be not only accurate but also comprehensible. Comprehensibility enables human inspection and understanding of the model, making it possible to trace why individual predictions are made. Since most high-accuracy techniques produce opaque models, accuracy is, in practice, regularly sacrificed for comprehensibility. One frequently studied technique, often able to reduce this accuracy vs. comprehensibility tradeoff, is rule extraction, i.e., the activity where another, transparent, model is generated from the opaque. In this paper, it is argued that techniques producing transparent models, either directly from the dataset, or from an opaque model, could benefit from using an oracle guide. In the experiments, genetic programming is used to evolve decision trees, and a neural network ensemble is used as the oracle guide. More specifically, the datasets used by the genetic programming when evolving the decision trees, consist of several different combinations of the original training data and "oracle data", i.e., training or test data instances, together with corresponding predictions from the oracle. In total, seven different ways of combining regular training data with oracle data were evaluated, and the results, obtained on 26 UCI datasets, clearly show that the use of an oracle guide improved the performance. As a matter of fact, trees evolved using training data only had the worst test set accuracy of all setups evaluated. Furthermore, statistical tests show that two setups, both using the oracle guide, produced significantly more accurate trees, compared to the setup using training data only.

A Survey of Evolutionary Algorithms for Decision-Tree Induction

“…Finally, the population can be a mixture of trees generated by either the full or the grow method, a procedure named ramped half and half [56]- [60]. Random initialization of decision trees encoded as trees is further discussed in [40], [61]- [64].…”

“…This approach or minor variations of it are found in [39], [60], [65], [66], [74]. Tsakonas and Dounias [75] also propose a weightedformula that combines accuracy and a simplicity component, though the idea is to penalize for smaller-sized trees due to domain constraints.…”

“…Another popular choice in EAs for decision tree induction is the roulette wheel selection.Works that implement this strategy for evolving decision trees are [35]- [37], [42]- [44], [48], [51], [55], [60]- [63], [73], [82], [86], [87].…”

“…Once again, there is no consensus on the most suitable value for this parameter. Values like 3% [28], [45], 10% [54], 20% [70], [71] and 25% [60], [65] of the number of generations have been used.…”

Using Feature Selection with Bagging and Rule Extraction in Drug Discovery

“…Despite this, all rule extraction algorithms that we are aware of use only training data (possibly with the addition of artificially generated instances) when extracting the transparent model. We have previously argued that it could be advantageous for a data miner to use test data input vectors together with predictions from the opaque model when performing rule extraction [10]. In this situation, the highly accurate opaque model, called the oracle since the target values it produces are treated as ground truth by the training regime of the transparent model, will coach the weaker transparent model.…”