NGBoost: Natural Gradient Boosting for Probabilistic Prediction

Abstract

We present Natural Gradient Boosting (NG-Boost), an algorithm which brings probabilistic prediction capability to gradient boosting in a generic way. Predictive uncertainty estimation is crucial in many applications such as healthcare and weather forecasting. Probabilistic prediction, which is the approach where the model outputs a full probability distribution over the entire outcome space, is a natural way to quantify those uncertainties. Gradient Boosting Machines have been widely successful in prediction tasks on structured input data, but a simple boosting solution for probabilistic prediction of real valued outputs is yet to be made. NGBoost is a gradient boosting approach which uses the Natural Gradient to address technical challenges that makes generic probabilistic prediction hard with existing gradient boosting methods. Our approach is modular with respect to the choice of base learner, probability distribution, and scoring rule. We show empirically on several regression datasets that NG-Boost provides competitive predictive performance of both uncertainty estimates and traditional metrics.

{\sc mirkwood:} Fast and Accurate SED Modeling Using Machine Learning

“…Our aim is to learn a data-driven mapping from the set of all x i s to the set of all y i s, to be able to predict y new for a new, unseen observation x new . To solve this regression problem 5 , we employ the NGBoost algorithm (Duan et al 2019). Unlike most commonly used ML algorithms and libraries such as Random Forests (Breiman 2001), Randomized Trees (Geurts et al 2006), XGBoost (Chen & Guestrin 2016) and Gradient Boosting Machines (Ke et al 2017), NGBoost enables us to easily work in a probabilistic setting, and corresponding to every input galaxy SED, output both a measure of the central tendency (i.e.…”

“…We make the assumption that samples are drawn from Gaussian distributions. Our loss function is the negative likelihood function (Duan et al 2019).…”

Short-Term Solar Irradiance Forecasting Using Calibrated Probabilistic Models

“…Network), and a decision-tree based model using natural gradient boosting (NGBoost) assuming a Gaussian output distribution [24]. Hyperparameters are provided in the appendix.…”

RoNGBa: A Robustly Optimized Natural Gradient Boosting Training Approach with Leaf Number Clipping

“…Our experiments use datasets from the UCI Machine Learning Repository, and follow the same protocol as NGBoost (Hernández-Lobato and Adams, 2015;Duan et al, 2019). For all datasets, we hold out a random 10% of the examples as a test set.…”

“…These implementations can train models with hundreds of trees using millions of training examples in a matter of minutes. NGBoost (Duan et al, 2019) generalized Natural Gradient as the direction of the steepest ascent in Riemannian space, and applied it for boosting to enable the probabilistic predication capability for the regression tasks. Natural gradient boosting shows promising performance improvements on small datasets due to better training dynamics, but it suffers from slow training speed overhead especially for large datasets.…”