End-to-End Learning of Decision Trees and Forests

Abstract

Conventional decision trees have a number of favorable properties, including a small computational footprint, interpretability, and the ability to learn from little training data. However, they lack a key quality that has helped fuel the deep learning revolution: that of being end-to-end trainable. Kontschieder et al. (ICCV, 2015) have addressed this deficit, but at the cost of losing a main attractive trait of decision trees: the fact that each sample is routed along a small subset of tree nodes only. We here present an end-to-end learning scheme for deterministic decision trees and decision forests. Thanks to a new model and expectation-maximization training scheme, the trees are fully probabilistic at train time, but after an annealing process become deterministic at test time. In experiments we explore the effect of annealing visually and quantitatively, and find that our method performs on par or superior to standard learning algorithms for oblique decision trees and forests. We further demonstrate on image datasets that our approach can learn more complex split functions than common oblique ones, and facilitates interpretability through spatial regularization.

One-Stage Tree: end-to-end tree builder and pruner

“…Hard Tree selects a subpath for instances according to a specific feature and threshold. In the multivariate tree (Irsoy et al, 2012;Norouzi et al, 2015;Irsoy et al, 2014;Hehn et al, 2019), which is also called soft tree, i is a continuous variable and s(x; i ) defines an oblique split.…”

“…Being aware of the benefits of discretization in terms of interpretability, End2End Tree (Hehn et al, 2019) proposes a multivariate discrete tree. End2End Tree is fully probabilistic at train time but becomes deterministic at test time after an annealing process.…”

“…Moreover, despite the low empirical error, decision trees are easily overfitted (Kotsiantis, 2013). To improve learning through end-to-end training with back-propagation, soft trees (Norouzi et al, 2015;Irsoy et al, 2014;Hehn et al, 2019) are proposed.…”

“…All of the above methods directly utilize probabilistic trees when testing, which looses the interpretability of decision trees. End2End Tree (Hehn et al, 2019) maintains a probabilistic tree for training and discretizes it to a deterministic one for testing. However, there exists a performance gap after the discretization.…”

Identification of Methamphetamine Abstainers by Resting-State Functional Magnetic Resonance Imaging

“…Since each person has multiple sliding windows, here we take the number of windows for every person in each class as the new features for classification. The decision tree is used as the classifier ( Douglas et al, 2011 ), where the subject searches along a single path from the root to the leaf, and the path depends on the characteristics of the sample ( Hehn et al, 2019 ). The decision tree can handle uneven data without standardizing and quantifying the data, and the logic is simple and intuitive.…”