An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation

Martinez-Muoz, G.; Hernández-Lobato, Daniel; Suárez, Alberto

doi:10.1109/tpami.2008.78

Cited by 271 publications

(159 citation statements)

References 37 publications

Supporting

Mentioning

149

Contrasting

Unclassified

Order By: Relevance

“…There are many subforest selection algorithms that prunes a number of decision trees from a decision forest while retaining or increasing ensemble accuracy (Adnan and Islam (2016c), Lu et al (2010), Margineantu and Dietterich (1997), Martínez-Muñoz et al (2009), Martínez-Muñoz and Suárez (2004), Ruta and Gabrys (2005)). It was shown in Adnan and Islam (2016c)) that if the number of trees in a subforest drops considerably, the ensemble accuracy also drops significantly.…”

Section: Australasian Journal Of Information Systems Adnan and Islam 20mentioning

confidence: 99%

ForEx++: A New Framework for Knowledge Discovery from Decision Forests

Adnan

Islam

2017

AJIS

View full text Add to dashboard Cite

Decision trees are popularly used in a wide range of real world problems for both prediction and classification (logic) rules discovery. A decision forest is an ensemble of decision trees and it is often built for achieving better predictive performance compared to a single decision tree. Besides improving predictive performance, a decision forest can be seen as a pool of logic rules (rules) with great potential for knowledge discovery. However, a standard-sized decision forest usually generates a large number of rules that a user may not able to manage for effective knowledge analysis. In this paper, we propose a new, data set independent framework for extracting those rules that are comparatively more accurate, generalized and concise than others. We apply the proposed framework on rules generated by two different decision forest algorithms from some publicly available medical related data sets on dementia and heart disease. We then compare the quality of rules extracted by the proposed framework with rules generated from a single J48 decision tree and rules extracted by another recent method. The results reported in this paper demonstrate the effectiveness of the proposed framework.

show abstract

Section: Australasian Journal Of Information Systems Adnan and Islam 20mentioning

confidence: 99%

ForEx++: A New Framework for Knowledge Discovery from Decision Forests

Adnan

Islam

2017

AJIS

View full text Add to dashboard Cite

show abstract

“…One motivation is that during this process a subset of models with uncorrelated models (or diverse models) can be selected, promoting the diversity in the ensemble. Several strategies have been employed to select the members for the ensemble, including Genetic Algorithms [21], Particle Swarm Optimization [28], Bayesian Artificial Immune System [22], and pruning strategies [17].…”

Section: Key Factors In Ensemble Systemsmentioning

confidence: 99%

“…A good way to alleviate this problem is the adequate selection of the subset of models from the original set of models [15,16]. This approach is also known as ensemble pruning [17]. The aim is to find a good subset of ensemble members in order to improve generalization ability, and which additionally reduces the system complexity.…”

Section: Introductionmentioning

confidence: 99%

Comparison of a genetic algorithm and simulated annealing for automatic neural network ensemble development

2013

View full text Add to dashboard Cite

In the last decades ensemble learning has established itself as a valuable strategy within the computational intelligence modeling and machine learning community. Ensemble learning is a paradigm where multiple models combine in some way their decisions, or their learning algorithms, or different data to improve the prediction performance. Ensemble learning aims at improving the generalization ability and the reliability of the system. Key factors of ensemble systems are diversity, training and combining ensemble members to improve the ensemble system performance. Since there is no unified procedure to address all these issues, this work proposes and compares Genetic Algorithm and Simulated Annealing based approaches for the automatic development of Neural Network Ensembles for regression problems. The main contribution of this work is the development of optimization techniques that select the best subset of models to be aggregated taking into account all the key factors of ensemble systems (e.g., diversity, training ensemble members and combination strategy). Experiments on two well-known data sets are reported to evaluate the effectiveness of the proposed methodologies. Results show that these outperform other approaches including Simple Bagging, Negative Correlation Learning (NCL), AdaBoost and GASEN in terms of generalization ability.

show abstract

“…While the boosting pruning problem was proven to be NP-complete by Tamon and Xiang (2000), several algorithms have been proposed to select a small set of hypotheses from a larger ensemble (see e.g. Martínez-Muñoz et al 2009;Tsoumakas et al 2009 for an overview). It is worth noting that most of the pruning algorithms are more effective when the ensembles are created by bagging rather than boosting, although the latter provides a natural ordering of the hypotheses (Martínez-Muñoz et al 2009).…”

Section: Related Workmentioning

confidence: 99%

“…Martínez-Muñoz et al 2009;Tsoumakas et al 2009 for an overview). It is worth noting that most of the pruning algorithms are more effective when the ensembles are created by bagging rather than boosting, although the latter provides a natural ordering of the hypotheses (Martínez-Muñoz et al 2009).…”

Section: Related Workmentioning

confidence: 99%

BoostingTree: parallel selection of weak learners in boosting, with application to ranking

2013

View full text Add to dashboard Cite

Boosting algorithms have been found successful in many areas of machine learning and, in particular, in ranking. For typical classes of weak learners used in boosting (such as decision stumps or trees), a large feature space can slow down the training, while a long sequence of weak hypotheses combined by boosting can result in a computationally expensive model. In this paper we propose a strategy that builds several sequences of weak hypotheses in parallel, and extends the ones that are likely to yield a good model. The weak hypothesis sequences are arranged in a boosting tree, and new weak hypotheses are added to promising nodes (both leaves and inner nodes) of the tree using some randomized method. Theoretical results show that the proposed algorithm asymptotically achieves the performance of the base boosting algorithm applied. Experiments are provided in ranking web documents and move ordering in chess, and the results indicate that the new strategy yields better performance when the length of the sequence is limited, and converges to similar performance as the original boosting algorithms otherwise.

show abstract

An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation

Abstract: Esta es la versión de autor del artículo publicado en: This is an author produced version of a paper published in: computationally more costly methods that directly select optimal or near-optimal subensembles.

Cited by 271 publications

References 37 publications

ForEx++: A New Framework for Knowledge Discovery from Decision Forests

ForEx++: A New Framework for Knowledge Discovery from Decision Forests

Comparison of a genetic algorithm and simulated annealing for automatic neural network ensemble development

BoostingTree: parallel selection of weak learners in boosting, with application to ranking

Contact Info

Product

Resources

About