Bayesian Regression Tree Ensembles that Adapt to Smoothness and Sparsity

Abstract

Summary Ensembles of decision trees are a useful tool for obtaining flexible estimates of regression functions. Examples of these methods include gradient‐boosted decision trees, random forests and Bayesian classification and regression trees. Two potential shortcomings of tree ensembles are their lack of smoothness and their vulnerability to the curse of dimensionality. We show that these issues can be overcome by instead considering sparsity inducing soft decision trees in which the decisions are treated as probabilistic. We implement this in the context of the Bayesian additive regression trees framework and illustrate its promising performance through testing on benchmark data sets. We provide strong theoretical support for our methodology by showing that the posterior distribution concentrates at the minimax rate (up to a logarithmic factor) for sparse functions and functions with additive structures in the high dimensional regime where the dimensionality of the covariate space is allowed to grow nearly exponentially in the sample size. Our method also adapts to the unknown smoothness and sparsity levels, and can be implemented by making minimal modifications to existing Bayesian additive regression tree algorithms.

BART with targeted smoothing: An analysis of patient-specific stillbirth risk

“…In our approach, by contrast, we smooth over just one target covariate. This avoids the high computational cost associated with the method of Linero and Yang (2017). Moreover, it avoids the somewhat inflexible interpolation and extrapolation behavior associated with their approach to global smoothing.…”

“…BART has been successful in a variety of contexts including prediction and classification (Chipman et al, 2010;Murray, 2017;Linero and Yang, 2017;Linero, 2018;Hernández et al, 2018), survival analysis (Sparapani et al, 2016), and causal inference (Hill, 2011;Hahn et al, 2017;Logan et al, 2017;Sivaganesan et al, 2017).…”

“…As in the original BART model, we scale the variance parameter τ 2 inversely by the number of trees m. Since the meanresponse function f (t; x) is the sum of m trees, this implies that the marginal prior variance of f (t; x) at any point t is τ 2 . We then assign τ a half-Cauchy prior as in Gelman (2006), Linero and Yang (2017) and Hahn et al (2017).…”

“…Smooth or partially smooth extensions of Bayesian tree models have also been proposed previously by Linero and Yang (2017), who smooth a regression tree ensemble by randomizing the decision rules at internal nodes of the tree. This model induces smoothness over all covariates by effectively replacing the step function induced by the binary trees with sigmoids.…”

Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects

“…There have been recent developments on posterior consistency and rates of posterior concentration for Bayesian tree models in prediction contexts (Linero and Yang, 2017;Rockova and van der Pas, 2017). These results require significant modifications to the BART prior, however, which we do not further investigate here.…”