Ensembles of Multi-Objective Decision Trees

Abstract

Abstract. Ensemble methods are able to improve the predictive performance of many base classifiers. Up till now, they have been applied to classifiers that predict a single target attribute. Given the non-trivial interactions that may occur among the different targets in multi-objective prediction tasks, it is unclear whether ensemble methods also improve the performance in this setting. In this paper, we consider two ensemble learning techniques, bagging and random forests, and apply them to multi-objective decision trees (MODTs), which are decision trees that predict multiple target attributes at once. We empirically investigate the performance of ensembles of MODTs. Our most important conclusions are: (1) ensembles of MODTs yield better predictive performance than MODTs, and (2) ensembles of MODTs are equally good, or better than ensembles of single-objective decision trees, i.e., a set of ensembles for each target. Moreover, ensembles of MODTs have smaller model size and are faster to learn than ensembles of single-objective decision trees.

Hierarchical annotation of medical images

“…From the curves we can note the increase of the predictive performance when we use ensembles instead of single tree. The lift in performance that ensembles give to their base classifier was previously noted in the cases of classification and regression [11,12] and multiple targets prediction [8]. The excellent performance for the prediction task for axes T and B (AUPRC of 0.9994 and 0.9862) is due to the simplicity of the problem.…”

Iterative classification for multiple target attributes

“…The second tested method is a state-of-the-art multi-target classification system, namely the ensembles of Multi-Objective Decision Trees (Kocev et al, 2007). Kocev et al (2007) study the performance of the popular Multi-Objective Decision Trees (MODTs) algorithm and conclude that ensembles of MODTs improve the predictive accuracy of the MODTs method.…”

“…Kocev et al (2007) study the performance of the popular Multi-Objective Decision Trees (MODTs) algorithm and conclude that ensembles of MODTs improve the predictive accuracy of the MODTs method. We obtained the settings of the ensembles of MODTs from their authors.…”

“…Following the same line of research but focusing on the well-known CN2 algorithm (Clark and Niblett, 1989), Zenko and Dzeroski (2008) proposed the Predictive Clustering Rules (PCRs) method for multitask problems. Inspired by the success of ensemble models, Kocev et al (2007) introduce the ensembles of multi-objective decision trees.…”

“…The computation scheme is developed as a wrapper in which many single-target classification algorithms can be simply "plugged-in" to simultaneously predict multiple target attributes. An empirical evaluation using eight data sets shows that the proposed method outperforms two benchmarking multi-task (-target) learning approaches, namely the multi-task neural networks (Caruana, 1997) and the ensembles of multi-objective decision trees (Kocev et al, 2007), as well as the approach that constructs independent single-target classifiers for each target attribute. Interestingly, our results also suggest that the iterative classification process improves the simultaneous prediction of the collection of independent classifiers by forcing these classifiers to make correct classifications on common instances instead of boosting their individual accuracies.…”