Minimising Decision Tree Size as Combinatorial Optimisation

Abstract

Decision tree induction techniques attempt to find small trees that fit a training set of data. This preference for smaller trees, which provides a learning bias, is often justified as being consistent with the principle of Occam's Razor. Informally, this principle states that one should prefer the simpler hypothesis. In this paper we take this principle to the extreme. Specifically, we formulate decision tree induction as a combinatorial optimisation problem in which the objective is to minimise the number of nodes in the tree. We study alternative formulations based on satisfiability, constraint programming, and hybrids with integer linear programming. We empirically compare our approaches against standard induction algorithms, showing that the decision trees we obtain can sometimes be less than half the size of those found by other greedy methods. Furthermore, our decision trees are competitive in terms of accuracy on a variety of well-known benchmarks, often being the most accurate. Even when post-pruning of greedy trees is used, our constraint-based approach is never dominated by any of the existing techniques.

Learning Optimal Classification Trees Using a Binary Linear Program Formulation

“…Hence, greedy based heuristics such as CART (Breiman et al 1984) and ID3 (Quinlan 1986) have been widely used to construct sub-optimal trees. Recent years have seen an increasing number of work that employ various Mathematical Optimization methods to build better quality decision trees, e.g., (Bennett and Blue 1996;Bessiere, Hebrard, and O'Sullivan 2009;Bertsimas and Dunn 2017;Silva 2017;Dash, Günlük, and Wei 2018;Blanquero et al 2018a;2018b;Firat et al 2018).…”

“…An advantage of these Mathematical Optimization based approaches is that they are able to employ the powerful optimization solvers to find decision trees. This power has led to interesting new approaches for learning models and rules, see e.g., (Bessiere, Hebrard, and O'Sullivan 2009;De Raedt, Guns, and Nijssen 2010;Narodytska et al 2018;Verwer, Zhang, and Ye 2017). In addition, the mathematical optimization models allow flexibility on modeling different learning objectives.…”

Constrained clustering by constraint programming

“…Constraint Programming has already been shown to be a promising approach for Data Mining through various tasks, such as itemset mining [40][41][42][43][44], skypattern mining [45] or decision tree construction [46].…”

Column generation based heuristic for learning classification trees

“…The objective function (1) maximizes the number of rows correctly predicted, that is, the accuracy of the decision tree. Constraint (2) imposes that exactly one path has to be selected for each leaf. Constraint (3) ensures that each row is directed to exactly one leaf.…”