Ensemble Pruning via Quadratic Margin Maximization

Abstract

Ensemble models refer to methods that combine a typically large number of classifiers into a compound prediction. The output of an ensemble method is the result of fitting a base-learning algorithm to a given data set, and obtaining diverse answers by reweighting the observations or by resampling them using a given probabilistic selection. A key challenge of using ensembles in large-scale multidimensional data lies in the complexity and the computational burden associated with them. The models created by ensembles are often difficult, if not impossible, to interpret and their implementation requires more computational power than single classifiers. Recent research effort in the field has concentrated in reducing ensemble size, while maintaining their predictive accuracy. We propose a method to prune an ensemble solution by optimizing its margin distribution, while increasing its diversity. The proposed algorithm results in an ensemble that uses only a fraction of the original classifiers, with improved or similar generalization performance. We analyze and test our method on both synthetic and real data sets. The simulations show that the proposed method compares favorably to the original ensemble solutions and to other existing ensemble pruning methodologies.

A Survey of Uncertainty in Deep Neural Networks

“…In Figure 4, the different underlying principles that are used to differentiate between the different types of methods are presented. • Original works 12 • Stochastic MCMC 13 • Theoretic Advances [20] 12 [78], [79], [80] 13 [81] 14 [82] 15 [83] 16 [63] 17 [84] 18 [31] 19 [85] 20 [86] 21 [87], [88], [89] 22 [90], [45] 23 [39] 24 [40] 25 [91] A. Single Deterministic Methods…”

A new correlation-based approach for ensemble selection in random forests

“…Furthermore, the discrete artificial fish swarm algorithm combined with margin distance minimization was applied by Zhu et al (2010), using a combination of diversity measure and heuristic algorithm to find the trade-off diversity and accuracy. For large-scale multidimensional data application, Martinez (2019) proposed an algorithm that produces a Ensemble selection in random forests reduced sub-ensemble of classifiers by optimizing the diversity and maximizing its lowermargin distribution.…”

Interpretable Deep Learning for Nonlinear System Identification Using Frequency Response Functions With Ensemble Uncertainty Quantification

“…Bayesian methods can use variational inference [35], [36], stochastic gradient MCMC [37], scalable weight averaging [38] and Monte Carlo dropout [39], [40]. Ensemble methods include deep ensembles based on random initialisations of the training process and shuffling data [41], sub-ensembles with weight sharing [42], and network pruning to reduce the ensemble size [43]. An appealing aspect of ensemble methods is that they tend to be simple to implement and require little tuning in terms of hyperparameters [41].…”