End-to-End Learning of Deterministic Decision Trees

Abstract

Conventional decision trees have a number of favorable properties, including interpretability, a small computational footprint and the ability to learn from little training data. However, they lack a key quality that has helped fuel the deep learning revolution: that of being end-to-end trainable, and to learn from scratch those features that best allow to solve a given supervised learning problem. Recent work (Kontschieder 2015) has addressed this deficit, but at the cost of losing a main attractive trait of decision trees: the fact that each sample is routed along a small subset of tree nodes only. We here propose a model and Expectation-Maximization training scheme for decision trees that are fully probabilistic at train time, but after a deterministic annealing process become deterministic at test time. We also analyze the learned oblique split parameters on image datasets and show that Neural Networks can be trained at each split node. In summary, we present the first endto-end learning scheme for deterministic decision trees and present results on par with or superior to published standard oblique decision tree algorithms.

ResOT: Resource-Efficient Oblique Trees for Neural Signal Classification

“…Therefore, only the most probable path needs to be evaluated at the test time. Similar results were reported in [25], [42] and our experimental results support this single-path inference scheme.…”

“…The training process for probabilistic trees is to maximize the above objective function. A detailed discussion on estimating the optimal parameters (θ * , ω * ) can be found in [25]. Intuitively, by maximizing Eq.…”

“…Unlike conventional axis-parallel trees that hold deterministic decision functions, probabilistic (i.e., soft) trees with oblique boundaries hold a probability decision leading to the left or right child. Inspired by back-propagation neural networks, such probabilistic trees with stochastic routing are compatible with gradient-based optimization [25]. As a result, one may effectively employ various model compression techniques such as fixed-point quantization [26], weight pruning, and sharing [27] that are widely used in hardware implementation of deep neural networks (DNNs).…”

End-to-End Learning of Decision Trees and Forests

“…This work extends our previous conference contribution (Hehn and Hamprecht 2018) where we introduced end-toend learning for decision trees. Here, we add an extension to decision forests, which we compare to state-of-the-art methods for training forests, and provide additional results on interpretability and the effect of the steepness parameter.…”

DecisioNet -- A Binary-Tree Structured Neural Network

“…Neural Trees With Hard Routing Another type of neural DT is proposed in [11,12], where the forward pass utilizes the routing nodes to make a hard decision, and in this way indeed only the relevant nodes of the tree are visited. [11] introduce a routing function which outputs binary values at test time only, such that the tree still performs soft decisions during training, allowing it to train end-to-end.…”