Fast Sparse Decision Tree Optimization via Reference Ensembles

Abstract

Sparse decision tree optimization has been one of the most fundamental problems in AI since its inception and is a challenge at the core of interpretable machine learning. Sparse decision tree optimization is computationally hard, and despite steady effort since the 1960's, breakthroughs have been made on the problem only within the past few years, primarily on the problem of finding optimal sparse decision trees. However, current state-of-the-art algorithms often require impractical amounts of computation time and memory to find optimal or near-optimal trees for some real-world datasets, particularly those having several continuous-valued features. Given that the search spaces of these decision tree optimization problems are massive, can we practically hope to find a sparse decision tree that competes in accuracy with a black box machine learning model? We address this problem via smart guessing strategies that can be applied to any optimal branch-and-bound-based decision tree algorithm. The guesses come from knowledge gleaned from black box models. We show that by using these guesses, we can reduce the run time by multiple orders of magnitude while providing bounds on how far the resulting trees can deviate from the black box's accuracy and expressive power. Our approach enables guesses about how to bin continuous features, the size of the tree, and lower bounds on the error for the optimal decision tree. Our experiments show that in many cases we can rapidly construct sparse decision trees that match the accuracy of black box models. To summarize: when you are having trouble optimizing, just guess.

Finding Interpretable Class-Specific Patterns through Efficient Neural Search

“…Rule-based classification (Lakkaraju, Bach, and Leskovec 2016;Dash, Gunluk, and Wei 2018;Chen and Rudin 2018;Proenc ¸a and van Leeuwen 2020;Hüllermeier, Fürnkranz, and Loza Mencia 2020;McTavish et al 2022;Huynh, Fürnkranz, and Beck 2023;Lin et al 2022) aims to find interpretable classification rules of the form if X 1 = 1∧X 5 = 1 then Y = 0. While such results are interpretable, these methods primarily focus on prediction rather than description and, hence, miss out on important details.…”

Exploring the Whole Rashomon Set of Sparse Decision Trees

“…Decision tree algorithms have a long history [17,18,19], but the vast majority of work on trees has used greedy induction [20,21] to avoid solving the NP-complete problem of finding an optimal tree [22]. However, greedy tree induction provides suboptimal trees, which has propelled research since the 1990s on mathematical optimization for finding optimal decision trees [4,23,24,25,26,27,28,29,30], as well as dynamic programming with branch-and-bound [31,32,33,34]. We refer readers to two recent reviews of this area [35,36].…”

“…Typically, the reference model is an empirical risk minimizer t ref ∈ arg min t∈trees Obj(t, x, y). Recent advances in decision tree optimization have allowed us to find this empirical risk minimizer, specifically using the GOSDT algorithm [31,32]. Our goal is to store R set ( , t ref , T ), sample from it, and compute statistics from it.…”

Quant-BnB: A Scalable Branch-and-Bound Method for Optimal Decision Trees with Continuous Features

“…[13] show useful computational gains by using pre-computed information from sub-trees and hash functions. [22] design smart guessing strategies to improve the performance of BnB-based approaches.…”

“…[27] report that a MIP solver cannot solve an optimal tree of depth 2 with less than 1000 observations and 10 features in 10 minutes. (ii) Most state-of-the-art algorithms [13,2,22] consider datasets with binary features (i.e, every feature is {0, 1}) rather than continuous ones. Algorithms for optimal trees with continuous features are much less developed 2 -our goal in this paper is to bridge this gap in the literature.…”