A Comparison of Ranking Methods for Classification Algorithm Selection

Abstract

We investigate the problem of using past performance information to select an algorithm for a given classification problem. We present three ranking methods for that purpose: average ranks, success rate ratios and significant wins. We also analyze the problem of evaluating and comparing these methods. The evaluation technique used is based on a leave-one-out procedure. On each iteration, the method generates a ranking using the results obtained by the algorithms on the training datasets. This ranking is then evaluated by calculating its distance from the ideal ranking built using the performance information on the test dataset. The distance measure adopted here, average correlation, is based on Spearman's rank correlation coefficient. To compare ranking methods, a combination of Friedman's test and Dunn's multiple comparison procedure is adopted. When applied to the methods presented here, these tests indicate that the success rate ratios and average ranks methods perform better than significant wins.

An Empirical Comparison of Weighting Functions for Multi-Label Distance Weighted K - Nearest Neighbour Method

“…In order to compare these methods comprehensively, we sort them using each measure first and then calculate the average rank over five measures for each method, as listed in the last rows of Tables 3-5. This comparison strategy was recommended in [18]. In Table 3 for the Emotions data, MLC-DWkNN-Dudani1 works the best on all five measures.…”

Smarter Classification for Imbalanced Data Set and Its Application to Patent Evaluation

“…Among the classification algorithms, the selection of the most adequate classification algorithm that fits to a specific problem is a difficult task (Brazdil and Soares, 2000). Training all classifiers to constitute an ensemble is a complex task and can lead to increased computational time and costs.…”

Discovering a taste for the unusual: exceptional models for preference mining

“…In the context of Label Ranking it is common to use the average ranking as the consensus ranking (Brazdil et al 2000). The average ranking is obtained by computing the average of the ranks, where the label with the lowest values is ranked in first place, and so on.…”