Predictive Distribution Modeling Using Transformation Forests

Abstract

Regression models for supervised learning problems with a continuous response are commonly understood as models for the conditional mean of the response given predictors. This notion is simple and therefore appealing for interpretation and visualization. Information about the whole underlying conditional distribution is, however, not available from these models. A more general understanding of regression models as models for conditional distributions allows much broader inference, for example, the computation of prediction intervals or probabilistic predictions for exceeding certain thresholds. Several random forest-type algorithms aim at estimating conditional distributions, most prominently quantile regression forests. We propose a novel approach based on a parametric family of distributions characterized by their transformation function. A dedicated novel "transformation tree" algorithm able to detect distributional changes is developed. Based on these transformation trees, we introduce "transformation forests" as an adaptive local likelihood estimator of conditional distribution functions. The resulting predictive distributions are fully parametric yet very general and allow inference procedures, such as likelihood-based variable importances, to be applied in a straightforward way. Supplemental files for this article are available online.

Distributional Gradient Boosting Machines

“…As splitting procedures, that are internally used to construct trees, can detect changes in the mean only, standard tree-based implementations are not able to recognize any distributional changes (e.g., change of variance), even if these can be related to covariates (Hothorn and Zeileis, 2021). As such, basic versions of XGBoost and LightGBM don't provide a way to model the full predictive distribution F Y (y|x), as they focus on predicting the conditional mean E(Y |X = x) only.…”

Flexible Predictive Distributions from Varying-Thresholds Modelling

“…There are strong links between the general model (3) and the so-called transformation models proposed by Hothorn et al (2014), Hothorn et al (2018), which have been extended to transformation forests more recently by Hothorn and Zeileis (2021). Transformation models assume P (Y ≤ y) = F (h(y)), where h(.)…”

“…is an unknown increasing transformation function. The transformation function can be additive (Hothorn et al, 2014) or nonparametric (Hothorn and Zeileis, 2021). Although the models look quite similar there are some crucial differences.…”

“…Quantile regression (Koenker and Hallock, 2001;Fitzenberger et al, 2013;Koenker et al, 2017Koenker et al, , 1994Meinshausen, 2006) allows to estimate the quantiles of the dependent variables as a function of explanatory variables. Transformation models estimate the whole predictive distributions under weak restrictions (Hothorn et al, 2014(Hothorn et al, , 2018Hothorn and Zeileis, 2021). The models proposed here also estimate the predictive distribution, but in contrast to transformation models the focus is on the dependence of thresholds on explanatory variables.…”

RFpredInterval: An R Package for Prediction Intervals with Random Forests and Boosted Forests

“…These approaches differ according to 1) the method used to build the forest and 2) the method used to build the prediction interval. There are four methods to build the forest, three from the classification and regression tree (CART) paradigm (Breiman and Breiman, 1984) and the transformation forest method (TRF) proposed by Hothorn and Zeileis (2021). Within the CART paradigm, in addition to the default least-squares (LS) splitting criterion, two alternative splitting criteria, L 1 and shortest prediction interval (SPI), are considered.…”