Regression Tree Analysis Using TARGET

Abstract

Regression trees are a popular alternative to classical regression methods. A number of approaches exist for constructing regression trees. Most of these techniques, including CART, are sequential in nature and locally optimal at each node split, so the final tree solution found may not be the best tree overall. In addition, small changes in the training data often lead to large changes in the final result due to the relative instability of these greedy tree-growing algorithms. Ensemble techniques, such as random forests, attempt to take advantage of this instability by growing a forest of trees from the data and averaging their predictions. The predictive performance is improved, but the simplicity of a single-tree solution is lost.In earlier work, we introduced the Tree Analysis with Randomly Generated and Evolved Trees (TARGET) method for constructing classification trees via genetic algorithms. In this article, we extend the TARGET approach to regression trees. Simulated data and real world data are used to illustrate the TARGET process and compare its performance to CART, Bayesian CART, and random forests. The empirical results indicate that TARGET regression trees have better predictive performance than recursive partitioning methods, such as CART, and single-tree stochastic search methods, such as Bayesian CART. The predictive performance of TARGET is slightly worse than that of ensemble methods, such as random forests, but the TARGET solutions are far more interpretable.

A Survey of Evolutionary Algorithms for Decision-Tree Induction

“…TARGET (Tree Analysis with Randomly Generated and Evolved Trees) [102], [103] is an EA that evolves regression trees where each candidate solution is an axis-parallel regression tree of variable size and shape. The initial population consists of 25 randomly created trees.…”

Evolutionary model tree induction

“…In addition, minor modifications in the training set often lead to large changes in the final model due to the intrinsic instability of these algorithms [9]. Ensemble methods were proposed to take advantage of these unstable algorithms by growing a forest of trees from the data and averaging their predictions.…”

“…TARGET [9], [16] is a GA proposed to evolve regression trees. It makes use of a Bayesian information criterion as the measure of tree fitness, which is basically a weighted-formula that penalizes for model complexity.…”

“…Model trees are traditionally induced by divide-and-conquer greedy algorithms which are sequential in nature and locally optimal at each node split [9]. Since inducing the best tree is a NP-Complete problem [23], a greedy heuristic may not derive the best overall tree.…”

An Introduction to Ensemble Methods for Data Analysis

“…This has led some researchers to develop algorithms in which a single "best" tree is constructed starting with a random sample of trees. However, such methods do not seem to perform as well as ensemble procedures (Fan and Gray, 2005), in part because overfitting remains a problem. Moreover, we will consider shortly alternative methods, within ensemble procedures, for showing how inputs are related to outputs.…”