How large should ensembles of classifiers be?

Abstract

Esta es la versión de autor del artículo publicado en: This is an author produced version of a paper published in: AbstractWe propose to determine the size of a parallel ensemble by estimating the minimum number of classifiers that are required to obtain stable aggregate predictions. Assuming that majority voting is used, a statistical description of the convergence of the ensemble prediction to its asymptotic (infinite size) limit is given. The analysis of the voting process shows that for most test instances the ensemble prediction stabilizes after only a few classifiers are polled. By contrast, a small but non-negligible fraction of these instances require large numbers of classifier queries to reach stable predictions. Specifically, the fraction of instances whose stable predictions require more than T classifiers for T ≫ 1 has a universal form and is proportional to T −1/2 . The ensemble size is determined as the minimum number of classifiers that are needed to estimate the infinite ensemble prediction at an average confidence level α, close to one. This approach differs from previous proposals, which are based on determining the size for which the prediction error (not the predictions themselves) stabilizes. In particular, it does not require estimates of the generalization performance of the ensemble, which can be unreliable. It has general validity because it is based solely on the statistical description of the convergence of majority voting to its asymptotic limit. Extensive experiments using representative parallel ensembles (bagging and random forest) illustrate the application of the proposed framework in a wide range of classification problems. These experiments show that the optimal ensemble size is very sensitive to the particular classification problem considered.

Random Rule Sets – Combining Random Covering with the Random Subspace Method

“…The results from [13,14] are not equally clear-cut and they indicate that different machine learning problems need different ensemble sizes. Both papers investigate how to dynamically create a dynamic stopping criterion to decide when to stop adding more classifiers to an ensemble.…”

Estimating a sharp convergence bound for randomized ensembles

“…In another direction, the paper [13] studied algorithmic convergence in terms of a different criterion, namely the "disagreement probability" δ t := P(M t (X) = M ∞ (X) D), where M ∞ (X) is the infinite ensemble analogue of the majority vote M t (X). In that work, an informal derivation is given to show that δ t is of order O( 1 √ t ).…”

“…[5,6,7,8,9,10,11] among others), comparatively little is known about how the prediction error depends on the number of classifiers (ensemble size). In particular, only a handful of works have considered this question from a theoretical standpoint [12,13,1,2] (cf. Section 1.2).…”

Estimating the algorithmic variance of randomized ensembles via the bootstrap

“…In the setting of classification, much of the literature has studied convergence in terms of the misclassification probability for majority voting, denoted err t (a counterpart of mse t ), which is viewed as a random variable that depends on ξ t and D. For this measure of error, the convergence of E[err t |D] and var(err t |D) as t → ∞ has been analyzed in the papers (Ng and Jordan, 2001;Lopes, 2016;Cannings and Samworth, 2017), which have developed asymptotic formulas for E[err t |D], as well as bounds for var(err t |D). Related results for a different measure of error can also be found in Hernández-Lobato et al (2013). More recently, our companion paper (Lopes, 2019) has developed a bootstrap method for measuring the convergence of err t , which is able to circumvent some of the limitations of analytical results.…”