An Evaluation of Over-Fit Control Strategies for Multi-Objective Evolutionary Optimization

Radtke, Paulo V. W.; Wong, Tony; Sabourin, Robert

doi:10.1109/ijcnn.2006.247331

Cited by 12 publications

(21 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The global validation strategy detailed in [10] selected better results in S than simply selecting the record solution r obtained at the end of the optimization process. This reinforces the conclusion in [10] that the optimization of classification systems using wrapped classifiers is prone to solution over-fit.…”

Section: Discussionmentioning

confidence: 99%

“…The RRT algorithm, detailed in Algorithm 1, produces after a number of iterations the record solution r. The algorithm is similar to a hill climbing approach, but avoids local optimum solutions by allowing the search towards non-optimal solutions with a fixed deviation D. Earlier experiments indicated that the RRT algorithm over-fitted solutions during the optimization process. The global validation strategy discussed in [10] is used to avoid this effect, and Algorithm 1 includes support for this strategy. Given the initial solution i, the algorithm will copy it to the record solution r and store its evaluation value in RECORD.…”

Section: Optimization Algorithmmentioning

confidence: 99%

“…EoCs are then created with K, producing the ensemble SE. To select the most accurate solution in the result set S, we use the global validation approach detailed in [10]. The full process is performed for both digits and uppercase letters.…”

Section: Experimental Protocolmentioning

confidence: 99%

See 2 more Smart Citations

Using the RRT algorithm to optimize classification systems for handwritten digits and letters

Radtke

Sabourin

Wong

2008

Proceedings of the 2008 ACM Symposium on Applied Computing

View full text Add to dashboard Cite

Multi-objective genetic algorithms have been often used to optimize classification systems, but little is discussed on their computational cost to solve such problems. This paper optimizes a classification system with an annealing based approach, the Record-toRecord Travel algorithm. Results obtained are compared to those obtained with a multi-objective genetic algorithm in the same approach. Experiments are performed with isolated handwritten digits and uppercase letters, demonstrating both the effectiveness and lower computational cost of the annealing based approach.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Optimization Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

Using the RRT algorithm to optimize classification systems for handwritten digits and letters

Radtke

Sabourin

Wong

2008

Proceedings of the 2008 ACM Symposium on Applied Computing

View full text Add to dashboard Cite

show abstract

“…There are three common methods for controlling overfitting in optimization systems [61]: Partial Validation (PV), Backwarding Validation (BV), and Global Validation (GV). In this work, we use the GV approach since previous works in the literature demonstrate that the GV is a more robust alternative for controlling overfitting in optimization techniques [49,61]. In the GV scheme (see Algorithm 1), at each generation, the fitness of all particles S g i ∈ S are evaluated using the validation set, DSEL * (line 18 of the algorithm).…”

Section: Overfitting Control Schemementioning

confidence: 99%

META-DES.Oracle: Meta-learning and feature selection for dynamic ensemble selection

2017

View full text Add to dashboard Cite

Dynamic ensemble selection (DES) techniques work by estimating the competence level of each classifier from a pool of classifiers, and selecting only the most competent ones for the classification of a specific test sample. The key issue in DES is defining a suitable criterion for calculating the classifiers' competence. There are several criteria available to measure the level of competence of base classifiers, such as local accuracy estimates and ranking. However, using only one criterion may lead to a poor estimation of the classifier's competence. In order to deal with this issue, we have proposed a novel dynamic ensemble selection framework using meta-learning, called META-DES. A meta-classifier is trained, based on the meta-features extracted from the training data, to estimate the level of competence of a classifier for the classification of a given query sample. An important aspect of the META-DES framework is that multiple criteria can be embedded in the system encoded as different sets of meta-features. However, some DES criteria are not suitable for every classification problem. For instance, local accuracy estimates may produce poor results when there is a high degree of overlap between the classes. Moreover, a higher classification accuracy can be obtained if the performance of the meta-classifier is optimized for the corresponding data. In this paper, we propose a novel version of the META-DES framework based on the formal definition of the Oracle, called META-DES.Oracle. The Oracle is an abstract method that represents an ideal classifier selection scheme. A meta-feature selection scheme using an overfitting cautious Binary Particle Swarm Optimization (BPSO) is proposed for improving the performance of the meta-classifier. The difference between the outputs obtained by the meta-classifier and those presented by the Oracle is minimized. Thus, the meta-classifier is expected to obtain results that are similar to the Oracle. Experiments carried out using 30 classifi-1 arXiv:1811.00217v1 [cs.LG] 1 Nov 2018 cation problems demonstrate that the optimization procedure based on the Oracle definition leads to a significant improvement in classification accuracy when compared to previous versions of the META-DES framework and other state-of-the-art DES techniques.

show abstract

“…Considering other methods applied on the same isolated digit dataset, we can find results varying from 99.16% to 99.37% [6,[21][22][23], using classifiers such as MLP and SVM, and ensembles of MLP as well, in a BL setting. In spite of HMMs performing worse than other classifiers in this task, some recent research with EoHMMs demonstrated that HMM-based classifiers, with improved codebooks, the recognition rates can be increased from 98.00% [8] to 98.86% [24].…”

Section: Experimental Evaluationmentioning

confidence: 99%

Evaluation of incremental learning algorithms for HMM in the recognition of alphanumeric characters

Cavalin

Sabourin

Suen

et al. 2009

Pattern Recognition

View full text Add to dashboard Cite

An Evaluation of Over-Fit Control Strategies for Multi-Objective Evolutionary Optimization

Cited by 12 publications

References 16 publications

Using the RRT algorithm to optimize classification systems for handwritten digits and letters

Using the RRT algorithm to optimize classification systems for handwritten digits and letters

META-DES.Oracle: Meta-learning and feature selection for dynamic ensemble selection

Evaluation of incremental learning algorithms for HMM in the recognition of alphanumeric characters

Contact Info

Product

Resources

About