Learning Optimal Decision Trees with MaxSAT and its Integration in AdaBoost

Hu, Hao; Siala, Mohamed; Hébrard, Emmanuel; Huguet, Marie‐José

doi:10.24963/ijcai.2020/163

Cited by 23 publications

(22 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many recent papers directly optimize the performance metric (e.g., accuracy) with soft or hard sparsity constraints on the tree size, where sparsity is measured by the number of leaves in the tree. Three major groups of these techniques are (1) mathematical programming, including mixed integer programming (MIP) [see the works of 28,29,251,301,302,118,6] and SAT solvers [214,130] [see also the review of 47], (2) stochastic search through the space of trees [e.g., 321,114,228], and (3) customized dynamic programming algorithms that incorporate branchand-bound techniques for reducing the size of the search space [131,185,222,78].…”

Section: Sparse Logical Models: Decision Trees Decision Lists Decision Setsmentioning

confidence: 99%

Interpretable machine learning: Fundamental principles and 10 grand challenges

Rudin¹,

Chen²,

Chen³

et al. 2022

Statist. Surv.

390

202

View full text Add to dashboard Cite

Interpretability in machine learning (ML) is crucial for high stakes decisions and troubleshooting. In this work, we provide fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic. We also identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem. Some of these problems are classically important, and some are recent problems that have arisen in the last few years. These problems are: (1) Optimizing sparse logical models such as decision trees;(2) Optimization of scoring systems; (3) Placing constraints into generalized additive models to encourage sparsity and better interpretability; (4) Modern case-based reasoning, including neural networks and matching for causal inference; (5) Complete supervised disentanglement of neural networks; (6) Complete or even partial unsupervised disentanglement of neural networks; (7) Dimensionality reduction for data visualization; (8) Machine learning models that can incorporate physics and other generative or causal constraints; (9) Characterization of the "Rashomon set" of good models; and (10) Interpretable reinforcement learning. This survey is suitable as a starting point for statisticians and computer scientists interested in working in interpretable machine learning.

show abstract

Section: Sparse Logical Models: Decision Trees Decision Lists Decision Setsmentioning

confidence: 99%

Interpretable machine learning: Fundamental principles and 10 grand challenges

Rudin¹,

Chen²,

Chen³

et al. 2022

Statist. Surv.

390

202

View full text Add to dashboard Cite

show abstract

“…Then, we evaluate the prediction performance between the proposed MaxSAT-BDD model and the heuristic methods, ODT and OODG (Kohavi and Li 1995). Next, we compare our MaxSAT-BDD model with an exact method for building decision trees using MaxSAT (Hu et al 2020) in terms of prediction quality, model size, and encoding size. Finally, we propose and evaluate a simple, yet efficient, scalable heuristic version of our MaxSAT-BDD model.…”

Section: Resultsmentioning

confidence: 99%

“…Compared to traditional approaches, exact methods offer guarantee of optimality, such as model size and accuracy. In this context, combinatorial optimisation methods, such as Constraint Programming (Bonfietti, Lombardi, and Milano 2015;Verhaeghe et al 2020), Mixed Integer Programming (Angelino et al 2018;Verwer and Zhang 2019;Aglin, Nijssen, and Schaus 2020), or Boolean Satisfiablility (SAT) (Bessiere, Hebrard, and O'Sullivan 2009;Narodytska et al 2018;Avellaneda 2020;Hu et al 2020;Janota and Morgado 2020;Yu et al 2020) have been successfully used to learn interpretable models. These declarative approaches are particularly interesting since they offer certain flexibility to handle additional requirements when learning a model.…”

Section: Introductionmentioning

confidence: 99%

“…As the BDD is ordered, this approach could not limit the number of features used, making it not quite comparable with our proposition. Another related work is in (Hu et al 2020) where the authors consider a MaxSAT model to learn optimal decision trees minimizing the classification error within a limited depth. The usage of the same solving methodology with the same objective function and the depth limit, makes these two MaxSAT models comparable.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Optimizing Binary Decision Diagrams with MaxSAT for classification

Hu,

Huguet,

Siala

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The growing interest in explainable artificial intelligence (XAI) for critical decision making motivates the need for interpretable machine learning (ML) models. In fact, due to their structure (especially with small sizes), these models are inherently understandable by humans. Recently, several exact methods for computing such models are proposed to overcome weaknesses of traditional heuristic methods by providing more compact models or better prediction quality. Despite their compressed representation of Boolean functions, Binary decision diagrams (BDDs) did not gain enough interest as other interpretable ML models. In this paper, we first propose SAT-based models for learning optimal BDDs (in terms of the number of features) that classify all input examples. Then, we lift the encoding to a MaxSAT model to learn optimal BDDs in limited depths, that maximize the number of examples correctly classified. Finally, we tackle the fragmentation problem by introducing a method to merge compatible subtrees for the BDDs found via the MaxSAT model. Our empirical study shows clear benefits of the proposed approach in terms of prediction quality and intrepretability (i.e., lighter size) compared to the state-of-the-art approaches.

show abstract

“…number of nodes, maximum/average depth, etc. [2,68,25,32,1,7,57,24,26,70,67,58,44,30,17,5,73,35,4,78,69,10,34,72,71,36,46,11,45] 2 . Moreover, there has been work on distilling or approximating complex ML models with (soft) decision trees [21,9,8,74,76,56,75].…”

Section: Introductionmentioning

confidence: 99%

On Explaining Decision Trees

Izza,

Ignatiev,

Marques-Silva

2020

Preprint

View full text Add to dashboard Cite

Decision trees (DTs) epitomize what have become to be known as interpretable machine learning (ML) models. This is informally motivated by paths in DTs being often much smaller than the total number of features. This paper shows that in some settings DTs can hardly be deemed interpretable, with paths in a DT being arbitrarily larger than a PI-explanation, i.e. a subset-minimal set of feature values that entails the prediction. As a result, the paper proposes a novel model for computing PI-explanations of DTs, which enables computing one PI-explanation in polynomial time. Moreover, it is shown that enumeration of PI-explanations can be reduced to the enumeration of minimal hitting sets. Experimental results were obtained on a wide range of publicly available datasets with well-known DT-learning tools, and confirm that in most cases DTs have paths that are proper supersets of PI-explanations.

show abstract

Learning Optimal Decision Trees with MaxSAT and its Integration in AdaBoost

Cited by 23 publications

References 12 publications

Interpretable machine learning: Fundamental principles and 10 grand challenges

Interpretable machine learning: Fundamental principles and 10 grand challenges

Optimizing Binary Decision Diagrams with MaxSAT for classification

On Explaining Decision Trees

Contact Info

Product

Resources

About