Risk Bounds for Embedded Variable Selection in Classification Trees

Abstract

The problems of model and variable selections for classification trees are jointly considered. A penalized criterion is proposed which explicitly takes into account the number of variables, and a risk bound inequality is provided for the tree classifier minimizing this criterion. This penalized criterion is compared to the one used during the pruning step of the CART algorithm. It is shown that the two criteria are similar under some specific margin assumptions. In practice, the tuning parameter of the CART penalty has to be calibrated by hold-out. Simulation studies are performed which confirm that the hold-out procedure mimics the form of the proposed penalized criterion.

Vapnik–Chervonenkis dimension of axis-parallel cuts

“…For example, some theoretical developments on dyadic partitions of R 2 are given in [4,1], and the VC dimension of axis-parallel cuts appears more particularly in the results obtained on the performance of classification and regression binary decision trees (CART) introduced by Breiman et. al [2] in 1984, and theoretically studied in [8,7,5,6]. In particular, it is to be found in the results of [6] that the VC dimension of axis-parallel cuts is of order log 2 d.…”

“…al [2] in 1984, and theoretically studied in [8,7,5,6]. In particular, it is to be found in the results of [6] that the VC dimension of axis-parallel cuts is of order log 2 d.…”

Risk bounds for CART classifiers under a margin condition

“…As for the regression case, the properties of the growing algorithm need to be analyzed to obtain full unconditional upper bounds. Results on the performance of theoretical procedures in which CART is viewed as a forward algorithm to approximate an ideal, but intractable, binary tree are given in [13]. Although they do not validate any concrete algorithm as done here, these results confirm that the penalty term used in penalized criterion ( 9) is well chosen under MA(1).…”

Discussion of ‘Parallel construction of decision trees with consistently non‐increasing expected number of tests’

“…Nevertheless, even if ENT is directly relied to the tree's complexity, taking only ENT into account means forgetting the important compromise to be made between misclassification rate and complexity. It is shown in Gey et al that this compromise is necessary to obtain decision trees of near optimal performance. The authors claim that the tree selected at the end of SF‐GOTA reaches this compromise, but with s ranging from 1 to at most 6, the trees provided by SF‐GOTA are not deep by construction.…”