Building diverse and optimized ensembles of gradient boosted trees for high-dimensional data

Abstract

Gradient Boosting Machines (GBMs) are powerful ensemble learning techniques that have been successfully applied to several low-dimensional applications. In GBMs, the learning algorithm sequentially fits new models to provide more accurate prediction of the response variable. Despite their high accuracy, GBMs suffer from major drawbacks such as high memory-consumption. In addition, given the fact that the learning algorithm is essentially sequential, it has problems with parallelization by design. Therefore, building optimized GBMs for high-dimensional applications requires powerful computations resources. In this paper, using real, high-dimensional (i.e. 1776 predictors) dataset, we demonstrate that by using different features selection/reduction techniques, the computations costs for building and tuning Tree-based GBMs can be substantially reduced at a slight drop in prediction accuracy. To cope with the data-intensive computations involved in building and tuning the ensembles, we utilize Amazon Elastic Compute Cloud (EC2) web service.