MurTree: Optimal Classification Trees via Dynamic Programming and Search

Abstract

Decision tree learning is a widely used approach in machine learning, favoured in applications that require concise and interpretable models. Heuristic methods are traditionally used to quickly produce models with reasonably high accuracy. A commonly criticised point, however, is that the resulting trees may not necessarily be the best representation of the data in terms of accuracy, size, and other considerations such as fairness. In recent years, this motivated the development of optimal classification tree algorithms that globally optimise the decision tree in contrast to heuristic methods that perform a sequence of locally optimal decisions. We follow this line of work and provide a novel algorithm for learning optimal classification trees based on dynamic programming and search. Our algorithm supports constraints on the depth of the tree and number of nodes and we argue it can be extended with other requirements. The success of our approach is attributed to a series of specialised techniques that exploit properties unique to classification trees. Whereas algorithms for optimal classification trees have traditionally been plagued by high runtimes and limited scalability, we show in a detailed experimental study that our approach uses only a fraction of the time required by the state-of-the-art and can handle datasets with tens of thousands of instances, providing several orders of magnitude improvements and notably contributing towards the practical realisation of optimal decision trees.

Optimal Decision Trees for Nonlinear Metrics

“…Benchmarks and experimental setting. We considered 75 binary classification datasets used in previous works (Verwer and Zhang 2019;Aglin, Nijssen, and Schaus 2020;Demirović et al 2020;Narodytska et al 2018;Hu, Rudin, and Seltzer 2019). The experiments were run one at a time on an Intel i7-3612QM@2.10 GHz with 8 GB RAM.…”

“…We describe previous work on dynamic programming and search for constructing decision trees with minimal misclassifications in more detail (Aglin, Nijssen, and Schaus 2020;Nijssen and Fromont 2007;Demirović et al 2020), as we generalise these methods.…”

“…Bounds by (Aglin, Nijssen, and Schaus 2020): (A) Given a parent node, its optimal left subtree with misclassification cost α, and the best incumbent for the parent node found so far is β misclassifications, then an upper bound of β −α−1 misclassifications may be imposed on the right subtree, and (B) if no tree with U B misclassifications could be found, then U B + 1 is stored in the cache as the lower bound for the subtree. Bounds by (Demirović et al 2020): (C) a lower bound for a node can be computed by considering every possible split and computing the sum of the lower bounds of the left and right subtrees and taking the minimum, and (D) similarity-based lower bounding computes a bound for a given dataset D 2 based on the optimal misclassification score α of another dataset D 1 using set difference operations. The intuition is as follows: optimistically assume that instances D 2 \D 1 will be perfectly classified and pessimistically assume that each instance from D 1 \ D 2 was misclassfied.…”

Robust and Heterogenous Odds Ratio: Estimating Price Sensitivity for Unbought Items

“…This is not the case of globally optimal trees, which are, on this respect, less interpretable (Bertsimas et al 2019a). Secondly, despite a vibrant corpus of works on formulating decision trees as mixed-integer optimization problems and solving them as such to provable optimality (Bertsimas and Dunn 2017, Günlük et al 2018, Zantedeschi et al 2020, Aglin et al 2020, Demirović et al 2020, Lin et al 2020, greedy procedures remain the gold-standard for balancing accuracy, interpretability, and scalability. Thirdly, recursive procedures can handle complex non-convex criteria like the Q-statistics.…”

On multivariate randomized classification trees: $l_0$-based sparsity, VC~dimension and decomposition methods

“…In [16] a column generation heuristic is described to build univariate binary classification trees for larger datasets. A dynamic programming and search algorithm is presented in [17] for constructing optimal univariate decision trees. In [18] the authors propose a flow-based MILO formulation for optimal univariate classification trees where they exploit the problem structure and max-flow/min-cut duality to derive a Benders' decomposition method for handling large datasets.…”