Why Cohen’s Kappa should be avoided as performance measure in classification

Abstract

We show that Cohen’s Kappa and Matthews Correlation Coefficient (MCC), both extended and contrasted measures of performance in multi-class classification, are correlated in most situations, albeit can differ in others. Indeed, although in the symmetric case both match, we consider different unbalanced situations in which Kappa exhibits an undesired behaviour, i.e. a worse classifier gets higher Kappa score, differing qualitatively from that of MCC. The debate about the incoherence in the behaviour of Kappa revolves around the convenience, or not, of using a relative metric, which makes the interpretation of its values difficult. We extend these concerns by showing that its pitfalls can go even further. Through experimentation, we present a novel approach to this topic. We carry on a comprehensive study that identifies an scenario in which the contradictory behaviour among MCC and Kappa emerges. Specifically, we find out that when there is a decrease to zero of the entropy of the elements out of the diagonal of the confusion matrix associated to a classifier, the discrepancy between Kappa and MCC rise, pointing to an anomalous performance of the former. We believe that this finding disables Kappa to be used in general as a performance measure to compare classifiers.

The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation

“…The current most popular and widespread metrics include Cohen’s kappa [70–72]: originally developed to test inter-rater reliability, in the last decades Cohen’s kappa entered the machine learning community for comparing classifiers’ performances. Despite its popularity, in the learning context there are a number of issues causing the kappa measure to produce unreliable results (for instance, its high sensitivity to the distribution of the marginal totals [73–75]), stimulating research for more reliable alternatives [76]. Due to these issues, we chose not to include Cohen’s kappa in the present comparison study.…”

Flood Hazard Risk Mapping Using a Pseudo Supervised Random Forest

“…Calculated indices are all based on four [75]. Due to recent concerns about Cohen's Kappa Coefficient assessments and its undesired behavior discussed in Reference [76], the MCC metric was also calculated to ensure the validity of our evaluation. MCC values close to 1 represent perfect agreement, the value of 0 is interpreted as random prediction and the value of −1 is interpreted as complete opposite predictions for observations.…”

Do simple syntactic heuristics to verb meaning hold up? Testing the structure mapping account over spontaneous speech to Spanish-learning children

“…Values close to −1 and +1 indicate performance much worse than chance, and much better than chance, respectively. As some concerns have been raised regarding the use of kappa as a performance measure in classification (Delgado and Tibau 2019), we also provide Matthew's Correlation Coefficients (MCC).…”