A hyper-heuristic evolutionary algorithm for automatically designing decision-tree algorithms

Abstract

Decision tree induction is one of the most employed methods to extract knowledge from data, since the representation of knowledge is very intuitive and easily understandable by humans. The most successful strategy for inducing decision trees, the greedy top-down approach, has been continuously improved by researchers over the years.This work, following recent breakthroughs in the automatic design of machine learning algorithms, proposes a hyper-heuristic evolutionary algorithm for automatically generating decision-tree induction algorithms, named HEAD-DT. We perform extensive experiments in 20 public data sets to assess the performance of HEAD-DT, and we compare it to the traditional decision-tree algorithms C4.5 and CART. Results show that HEAD-DT can generate algorithms that significantly outperform C4.5 and CART regarding predictive accuracy and F-Measure.

Contrasting meta-learning and hyper-heuristic research: the role of evolutionary algorithms

“…In the case of evolutionary artificial neural networks, researchers have gone one step further, and are using evolutionary algorithms to create ensembles of these networks [28], which is categorized as a combination method in meta-learning. After neural networks, approaches for building rule induction algorithms [87] and decision trees [9] were also proposed. These approaches resemble work in metaevolutionary algorithms, and in this case the outer loop algorithm is an evolutionary approach and the inner loop is essentially a learning algorithm, which can be, again, an evolutionary one.…”

“…A few years after this first work, [9] proposed to generate a complete algorithm to create a decision tree induction algorithm from sub-components of well-known algorithms. This was achieved using a genetic algorithm with a linear individual representation where each gene value represents a specific choice in the design of one component of a decision tree algorithm.…”

“…Finally, when operating in level C, also named meta-generator level, machine learning methods act as metalearning systems, generating a classifier generator. The meta-generator level is a recent development, and we are aware of only two meta-generator level systems [9,86,87].…”

“…Recent application of genetic programming in classification has resulted in a meta-learning system which can automatically generate new classification algorithms [9,87], which are in turn used to generate classifiers (classification models for the input data). This system operates at level C in Fig.…”

Investigating fitness functions for a hyper-heuristic evolutionary algorithm in the context of balanced and imbalanced data classification

“…HEAD-DT, which is subject of this paper, is capable of performing quite well when generating a novel decision-tree induction algorithm for a particular problem (data set) [2], and also for a group of data sets [4][5][6]. Nevertheless, HEAD-DT was employed optimizing always the same objective function (F-Measure), and, moreover, no consistent investigation was performed regarding the ability of HEAD-DT for dealing with data sets that share a particular structural characteristic.…”