A review of tree-based Bayesian methods

Abstract

Tree-based regression and classification ensembles form a standard part of the data-science toolkit. Many commonly used methods take an algorithmic view, proposing greedy methods for constructing decision trees; examples include the classification and regression trees algorithm, boosted decision trees, and random forests. Recent history has seen a surge of interest in Bayesian techniques for constructing decision tree ensembles, with these methods frequently outperforming their algorithmic counterparts. The goal of this article is to survey the landscape surrounding Bayesian decision tree methods, and to discuss recent modeling and computational developments. We provide connections between Bayesian tree-based methods and existing machine learning techniques, and outline several recent theoretical developments establishing frequentist consistency and rates of convergence for the posterior distribution. The methodology we present is applicable for a wide variety of statistical tasks including regression, classification, modeling of count data, and many others. We illustrate the methodology on both simulated and real datasets.

BARP: Improving Mister P Using Bayesian Additive Regression Trees

“…Although these methods are not a silver bullet, I demonstrate that they can do more with less. In this letter, I focus on Bayesian additive regression trees (BART or, when combined with post-stratification, BARP) because of its well-documented predictive capabilities (see Chipman, George, and McCulloch 2010; Linero 2017), intuitive connection with the post-stratification stage, and rich descriptive results on covariate importance and partial dependence. I compare BARP’s performance with several alternative regularization methods in Section 7 of the Supporting Information, finding that BARP is consistently the best-in-class method in terms of accuracy and is among the best methods in terms of correlation across geographic units.…”

Semiparametric mixed‐scale models using shared Bayesian forests

“…| 3 regression tree methods, see Chipman et al (2013) and Linero (2017). A fact which will be useful for specifying priors later and for making connections with other approaches is that under the conditions…”

Expected hypothetical completion probability

“…On its face, this assumption is tenuous at best, motivating us to consider estimating the unknown log-odds function f with regression trees, which naturally incorporate interactions by design. Specifically, we use Similar to the simple linear-logistic model above, to use BART we start by specifying a prior π(f ) mean to reflect all of our initial uncertainty about the unknown function f. In the case of BART, rather than specifying this prior directly, we instead specify a prior over the space of regression trees used to approximate f.. We then update this prior to compute a posterior over the space of regression trees, which induces a posterior over f. For a review of Bayesian tree-based methods, please see Linero (2017) for further details about the BART prior and Gibbs sampler, please see Chipman et al (2010). We fit the BART model using the lbart() function available in the "BART" R package (McCulloch et al, 2018).…”