Estimating a sharp convergence bound for randomized ensembles

Abstract

When randomized ensembles such as bagging or random forests are used for binary classification, the prediction error of the ensemble tends to decrease and stabilize as the number of classifiers increases. However, the precise relationship between prediction error and ensemble size is unknown in practice. In the standard case when classifiers are aggregated by majority vote, the present work offers a way to quantify this convergence in terms of "algorithmic variance," i.e. the variance of prediction error due only to the randomized training algorithm. Specifically, we study a theoretical upper bound on this variance, and show that it is sharp -in the sense that it is attained by a specific family of randomized classifiers. Next, we address the problem of estimating the unknown value of the bound, which leads to a unique twist on the classical problem of non-parametric density estimation. In particular, we develop an estimator for the bound and show that its MSE matches optimal non-parametric rates under certain conditions. (Concurrent with this work, some closely related results have also been considered in Cannings and Samworth [1] and Lopes [2].) melopes@ucdavis.edu.

Statistical optimality conditions for compressive ensembles

“…The third term in the bound corresponds to the error contribution from working with a finite ensemble size m. Note that the bound is non-asymptotic and holds for any finite ensemble size m, and one may set m to be of order n. The excess risk guarantee improves as the ensemble size m grows, at a speed that matches the asymptotically optimal rate m −1 determined in recent work of Lopes (2020).…”

“…However, we are interested in high probability bounds to reveal more information about the worst case behaviour for the excess risk, subject to a failure probability. In another line of research, recent work by Lopes (2020) determined the asymptotic speed of convergence as the number of predictors in the ensemble grows. While this is informative for very large ensembles, we are interested in non-asymptotic guarantees for ensembles of any given finite size.…”

Random projections: Data perturbation for classification problems

“…There is also some recent general theoretical work on understanding ensemble methods. Lopes (2019b) derives the rate at which the test error of a finite ensemble approaches its infinite simulation counterpart. Lopes (2019a) proposes a bootstrap method to approximate the variance of an ensemble, with a view to ascertain how large an ensemble is needed.…”

Domain Adaptive Bootstrap Aggregating

“…Algorithmic convergence has been studied for randomized ensembles to analyze the effect of the ensemble size B on prediction error; see Cannings and Samworth [2017] and Lopes [2020]. Define…”

“…Rbagging is essentially a weighted randomized ensemble method. We further quantify the convergence in terms of the algorithmic variance leveraging on recent results in Cannings and Samworth [2017] and Lopes [2020], showing that the algorithmic variance decreases as the number of bootstrap samples increases.…”