Multiobjective optimization of classifiers by means of 3D convex-hull-based evolutionary algorithms

Abstract

Finding a good classifier is a multiobjective optimization problem with different error rates and the costs to be minimized. The receiver operating characteristic is widely used in the machine learning community to analyze the performance of parametric classifiers or sets of Pareto optimal classifiers. In order to directly compare two sets of classifiers the area (or volume) under the convex hull can be used as a scalar indicator for the performance of a set of classifiers in receiver operating characteristic space.Recently, the convex hull based multiobjective genetic programming algorithm was proposed and successfully applied to maximize the convex hull area for binary classification problems. The contribution of this paper is to extend this algorithm for dealing with higher dimensional problem formulations. In particular, we discuss problems where parsimony (or classifier complexity) is stated as a third objective and multi-class classification with three different true classification rates to be maximized.The design of the algorithm proposed in this paper is inspired by indicator-based evolutionary algorithms, where first a performance indicator for a solution set is established and then a selection operator is designed that complies with the performance indicator. In this case, the performance indicator will be the volume under the convex hull. The algorithm is tested and analyzed in a proof of concept study on different benchmarks that are designed for measuring its capability to capture relevant parts of a convex hull.Further benchmark and application studies on email classification and feature selection round up the analysis and assess robustness and usefulness of the new algorithm in real world settings.

A spam filtering multi-objective optimization study covering parsimony maximization and three-way classification

“…In fact, they usually require achieving improvements on several conflicting goals, such as recall/sensibility, precision/specificity and classifier complexity [21], simultaneously. Apart from this classical perspective, many other examples of multi-objective optimization of ML classifiers can also be considered, such as the trade-off between learning new information and/or forgetting old one, or between learning as many details as possible and generalizing the model to its maximum in pattern recognition [30].…”

“…is currently being widely used by the scientific community, being able to represent the convex hull area of a set of points, each of which stands for an optimal classifier [21].…”

Quadcriteria Optimization of Binary Classifiers: Error Rates, Coverage, and Complexity

“…First, other multiobjective and many-objectives optimization algorithms of high potential for the anti-spam filtering type of problems will be studied and explored (e.g. MOEA/D and NSGA-III), and also tailor made approaches for classification such as CH-EMOA [1,8] or mixed integer optimization [2]. Secondly, analysis of the rules that reveal highest contributions for the classification process will be addressed, in order to assess not only quantitative classifier complexity, but also to explore the nature of the rules most frequently present in the best solutions.…”

“…However, as discussed in [8,14], binary classification of instances as positive or negative is sometimes too strict and can result in high misclassification costs. Three-way classification can also leave an email unclassified in case of low confidence in classification.…”

“…This way the maximization of coverage is formulated as the minimization of the number of unclassified samples. -Minimize complexity: In previous work on anti-spam filter optimization [8], it was observed that many rules were not participating in the classification process and those with very small scores only marginally influenced the classification results. This observation suggests that in addition to minimizing the occurrence of false positives and false negatives, the complexity of the anti-spam filter (or its parsimony) can also be optimized.…”