Solution Over-Fit Control in Evolutionary Multiobjective Optimization of Pattern Classification Systems

Abstract

The optimization of many engineering systems is challenged by the solution over-fit to the data set used to evaluate potential solutions during the evolutionary process. The solution over-fit phenomenon is hard to detect and is especially prevalent in problems involving example-based training, such as pattern feature selection and pattern classifier design. For these applications, uncontrolled over-fit can lead to biased features being extracted and degraded classifier generalization abilities. This paper details the performance of a solution over-fit control strategy used in the multiobjective evolutionary optimization of a multileveled classification system. This control, embedded within a solution validation procedure, minimizes the over-fit effects without modifying the dominance relation used in the processing of candidate solutions. Extensive experimental analysis using multiobjective genetic and memetic algorithms demonstrates both the need and the efficiency of the proposed over-fit control for pattern classification systems optimization.

Overfitting cautious selection of classifier ensembles with genetic algorithms

“…Several search algorithms have been applied in the literature for classifier selection, ranging from ranking the n best classifiers [19] to genetic algorithms (GAs) [28,33]. Ensemble combination performance [27], diversity measures [33,1,30] and ensemble size [23] are search criteria which are often employed. GAs are attractive since they allow the fairly easy implementation of classifier selection tasks as optimization processes [32].…”

“…Tsymbal et al [34] suggested that using individual member accuracy (instead of ensemble accuracy) together with diversity in a genetic search can overcome overfitting. Radtke et al [23] proposed a global validation method for multi-objective evolutionary optimization including classifier ensemble selection.…”

“…Llorà et al [14] suggested the use of optimization constraints to remove the overfitted solutions over the Pareto front in an evolutionary multi-objective learning system. Wiegand et al [35] [23] proposed global validation strategies for the evolutionary optimization of neural network parameters. Loughrey and Cunningham [15] presented an early-stopping criterion to control overfitting in wrapper-based feature subset selection using stochastic search algorithms such as GA and Simulated Annealing.…”

“…a set of nondominated solutions) is involved due to the fact that comparing sets of equal importance is a very complex task. Finally, the third contains the archive-based methods [23,35,26], which have been shown to be efficient tools for tackling overfitting [35,26,29] and may be widely reused in all optimization problems [23].…”

“…Since both single-objective GA and multi-objective GA (MOGA) are examined in this paper, we investigate the use of an auxiliary archive A to store the best performing candidate ensembles (or Pareto fronts in the MOGA case) obtained in a validation process using the validation partition V to control overfitting. Three different strategies for update A have been compared and adapted in this paper to the context of the single-and multiobjective selection of classifier ensembles: (1) partial validation where A is updated only in the last generation of the optimization process; (2) backwarding [26] which relies on monitoring the optimization process by updating A with the best solution from each generation; and (3) global validation [23] updating A by storing in it the Pareto front (or the best solution in the GA case) identified on V at each generation step.…”