Cost-conscious comparison of supervised learning algorithms over multiple data sets

Ulas, Aydin; Yıldız, Olcay Taner; Alpaydın, Ethem

doi:10.1016/j.patcog.2011.10.005

Cited by 41 publications

(16 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The number of runs is reduced to 51 (as suggested in Liang et al 2013). The statistical significance of pairwise comparisons among all 14 algorithms is tested by means of the Friedman's test with post hoc Shaffer's static procedure (Shaffer 1986), as suggested in Garcia and Herrera (2008) and Ulas et al (2012). The MATLAB implementation of Shaffer's procedure, developed by Ulas et al (2012), and available at www.cmpe.boun.edu.tr/~ulas/m2test/details.php, is applied in this study.…”

Section: Methods For Structural Bias Detectionmentioning

confidence: 99%

Searching for structural bias in particle swarm optimization and differential evolution algorithms

Piotrowski

Napiorkowski

2016

Swarm Intell

View full text Add to dashboard Cite

During the last two decades, a large number of metaheuristics have been proposed, leading to various studies that call for a deeper insight into the behaviour, efficiency and effectiveness of such methods. Among numerous concerns that are briefly reviewed in this paper, the presence of a structural bias (i.e. the tendency, not justified by the fitness landscape, to visit some regions of the search space more frequently than other regions) has recently been detected in simple versions of the genetic algorithm and particle swarm optimization. As of today, it remains unclear how frequently such a behaviour occurs in population-based swarm intelligence and evolutionary computation methods, and to what extent structural bias affects their performance. The present study focuses on the search for structural bias in various variants of particle swarm optimization and differential evolution algorithms, as well as in the traditional direct search methods proposed by Nelder-Mead and Rosenbrock half a century ago. We found that these historical direct search methods are structurally unbiased. However, most tested new metaheuristics are structurally biased, and at least some presence of structural bias can be observed in almost all their variants. The presence of structural bias seems to be stronger in particle swarm optimization algorithms than in differential evolution algorithms. The relationships between the strength of the structural bias and the dimensionality of the search space, the number of allowed function calls and the population size are complex and hard to generalize. For 14 algorithms tested on the CEC2011 realworld problems and the CEC2014 artificial benchmarks, no clear relationship between the strength of the structural bias and the performance of the algorithm was found. However, at least for artificial benchmarks, such old and structurally unbiased methods like Nelder-Mead algorithm performed relatively well. This is a warning that the presence of structural bias in novel metaheuristics may hamper their search abilities.

show abstract

Section: Methods For Structural Bias Detectionmentioning

confidence: 99%

Searching for structural bias in particle swarm optimization and differential evolution algorithms

Piotrowski

Napiorkowski

2016

Swarm Intell

View full text Add to dashboard Cite

show abstract

“…For the significance tests, the procedure suggested by Demsar [5] is utilized by using the Friedman significance test with the Nemenyi post-hoc procedure, with p < 0.05. The toolbox used in the implementation is provided in [15]. Here, the ensembles pruned by AcEc − P with the given pruning rates are assigned 0 if there is no statistical difference between them and the unpruned ensemble, and 1 otherwise.…”

Section: Discussion Of the Resultsmentioning

confidence: 99%

ECOC Matrix Pruning Using Accuracy Information

Zor

Windeatt

Kittler

2013

Multiple Classifier Systems

View full text Add to dashboard Cite

Abstract. The target of ensemble pruning is to increase efficiency by reducing the ensemble size of a multi classifier system and thus computational and storage costs, without sacrificing and preferably enhancing the generalization performance. However, most state-of-the-art ensemble pruning methods are based on unweighted or weighted voting ensembles; and their extensions to the Error Correcting Output Coding (ECOC) framework is not strongly evident or successful. In this study, a novel strategy for pruning ECOC ensembles which is based on a novel accuracy measure is presented. The measure is defined by establishing the link between the accuracies of the two-class base classifiers in the context of the main multiclass problem. The results show that the method outperforms the ECOC extensions of the state-of-the-art pruning methods in the majority of cases and that it is even possible to improve the generalization performance by only using 30% of the initial ensemble size in certain scenarios.

show abstract

“…As there are many different kernels for SVM available it is important to choose a suitable one, in accordance with the requirements of the data, in order to achieve the best classification results. A more complex kernel adds to the computation cost of the algorithm compared to a linear SVM (Ulaş et al 2012). However, given the recent development in ICT, most optimization problems of high-dimensionality, even…”

Section: Theoretical Background 1 Svm As a Classification Technique Hmentioning

confidence: 99%

Identifying Product and Process State Drivers in Manufacturing Systems Using Supervised Machine Learning

Wuest¹

2015

Springer Theses

View full text Add to dashboard Cite

Cost-conscious comparison of supervised learning algorithms over multiple data sets

Cited by 41 publications

References 19 publications

Searching for structural bias in particle swarm optimization and differential evolution algorithms

Searching for structural bias in particle swarm optimization and differential evolution algorithms

ECOC Matrix Pruning Using Accuracy Information

Identifying Product and Process State Drivers in Manufacturing Systems Using Supervised Machine Learning

Contact Info

Product

Resources

About