Optimal dyadic decision trees

Abstract

We introduce a new algorithm building an optimal dyadic decision tree (ODT). The method combines guaranteed performance in the learning theoretical sense and optimal search from the algorithmic point of view. Furthermore it inherits the explanatory power of tree approaches, while improving performance over classical approaches such as CART/C4.5, as shown on experiments on artificial and benchmark data.

Algorithms for optimal dyadic decision trees

“…4 Algorithms for λ Blanchard et al (2007) suggest that the regularization parameter take the form λ = κ/n, and they show good empirical results for 1 ≤ κ ≤ 4. They also suggest a rule of thumb that chooses κ = 2 for a 2-class classification problem (Blanchard et al 2007).…”

“…trees whose splits are determined by cycling through the coordinates and splitting at the interval mid-points) is guaranteed to be robust to distribution (Scott and Nowak 2003, 2004, 2006. Recently it has been shown that allowing the dyadic splits to be performed in arbitrary order, and then designing the tree to minimize a regularized risk, also yields robust performance guarantees (Scott and Nowak 2006;Blanchard et al 2007) and tends to give better results in practice (Blanchard et al 2007). The current best algorithm for designing these trees is the dynamic programming algorithm of Blanchard et al (2007) which was inspired by the "dyadic CART" algorithm of Donoho (1997).…”

“…Recently it has been shown that allowing the dyadic splits to be performed in arbitrary order, and then designing the tree to minimize a regularized risk, also yields robust performance guarantees (Scott and Nowak 2006;Blanchard et al 2007) and tends to give better results in practice (Blanchard et al 2007). The current best algorithm for designing these trees is the dynamic programming algorithm of Blanchard et al (2007) which was inspired by the "dyadic CART" algorithm of Donoho (1997). Blanchard et al provide a thorough development of this algorithm and its properties, and demonstrate its utility through a series of empirical experiments.…”

“…To solve this problem we follow the approach of Blanchard et al (2007) which chooseŝ T X to minimize a regularized empirical risk function. Our development requires that we decompose the problem into operations over dyadically constructed subsets of X.…”

“…Nevertheless this algorithm has proven very effective, e.g. it typically produces smaller and more accurate decision trees than C4.5 (Blanchard et al 2007). We develop a simple recursive algorithm whose run time (and memory usage) obeys the same worst case upper bound, but possesses a much smaller (polynomial-time) lower bound.…”