Consistency of random forests

Abstract

Random forests are a learning algorithm proposed by Breiman [Mach. Learn. 45 (2001) 5-32] that combines several randomized decision trees and aggregates their predictions by averaging. Despite its wide usage and outstanding practical performance, little is known about the mathematical properties of the procedure. This disparity between theory and practice originates in the difficulty to simultaneously analyze both the randomization process and the highly datadependent tree structure. In the present paper, we take a step forward in forest exploration by proving a consistency result for Breiman's [Mach. Learn. 45 (2001) 5-32] original algorithm in the context of additive regression models. Our analysis also sheds an interesting light on how random forests can nicely adapt to sparsity.

Estimation and Inference of Heterogeneous Treatment Effects using Random Forests

“…Recent results by Biau [2012], Meinshausen [2006], Mentch and Hooker [2016], Scornet et al [2015], and others have established asymptotic properties of particular variants and simplifications of the random forest algorithm. To our knowledge, however, we provide the first set of conditions under which predictions made by random forests are both asymptotically unbiased and Gaussian, thus allowing for classical statistical inference; the extension to the causal forests proposed in this paper is also new.…”

“…The convergence and consistency properties of trees and random forests have been studied by, among others, Biau [2012], Biau et al [2008], Breiman [2004], Breiman et al [1984], Meinshausen [2006], Scornet et al [2015], Wager and Walther [2015], and Zhu et al [2015]. Meanwhile, their sampling variability has been analyzed by Duan [2011], Lin and Jeon [2006], Mentch and Hooker [2016], Sexton andLaake [2009], and.…”

“…As observed by Buja and Stuetzle [2006], Chen and Hall [2003], Friedman and Hall [2007] and others, estimators of this form can exhibit surprising properties even in simple situations; meanwhile, using subsampling rather than bootstrap sampling has been found to avoid several pitfalls [e.g., Politis et al, 1999]. Although they are less common in the literature, random forests based on subsampling have also been occasionally studied and found to have good practical and theoretical properties [e.g., Bühlmann and Yu, 2002, Mentch and Hooker, 2016, Scornet et al, 2015, Strobl et al, 2007.…”

“…Following Breiman [2001a], almost all practical implementations of random forests are not honest. Moreover, there has been a stream of recent work providing theoretical guarantees for adaptive random forests: Scornet et al [2015] establish risk consistency under assumptions on the data-generating function, i.e., they show that Figure 3: Comparison of the performance of honest and adaptive causal forests when predicting at x 0 = (0, 0, . .…”

“…We note that there have been some recent theoretical investigations of non-honest forests, including Scornet et al [2015] and Wager and Walther [2015]. However, Scornet et al [2015] do not consider pointwise properties of forests; whereas Wager and Walther [2015] show consistency of adaptive forests with larger leaves, but their bias bounds decay slower than the sampling variance of the forests and so cannot be used to establish centered asymptotic normality.…”