The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification

Abstract

Binary classification is a common task for which machine learning and computational statistics are used, and the area under the receiver operating characteristic curve (ROC AUC) has become the common standard metric to evaluate binary classifications in most scientific fields. The ROC curve has true positive rate (also called sensitivity or recall) on the y axis and false positive rate on the x axis, and the ROC AUC can range from 0 (worst result) to 1 (perfect result). The ROC AUC, however, has several flaws and drawbacks. This score is generated including predictions that obtained insufficient sensitivity and specificity, and moreover it does not say anything about positive predictive value (also known as precision) nor negative predictive value (NPV) obtained by the classifier, therefore potentially generating inflated overoptimistic results. Since it is common to include ROC AUC alone without precision and negative predictive value, a researcher might erroneously conclude that their classification was successful. Furthermore, a given point in the ROC space does not identify a single confusion matrix nor a group of matrices sharing the same MCC value. Indeed, a given (sensitivity, specificity) pair can cover a broad MCC range, which casts doubts on the reliability of ROC AUC as a performance measure. In contrast, the Matthews correlation coefficient (MCC) generates a high score in its $$[-1; +1]$$ [ - 1 ; + 1 ] interval only if the classifier scored a high value for all the four basic rates of the confusion matrix: sensitivity, specificity, precision, and negative predictive value. A high MCC (for example, MCC $$=$$ = 0.9), moreover, always corresponds to a high ROC AUC, and not vice versa. In this short study, we explain why the Matthews correlation coefficient should replace the ROC AUC as standard statistic in all the scientific studies involving a binary classification, in all scientific fields.

Performance and Agreement When Annotating Chest X-ray Text Reports—A Preliminary Step in the Development of a Deep Learning-Based Prioritization and Detection System

“…Generally, all annotators showed high correlation [ 20 ] to “gold standard” annotations of CXR text reports ( Table 2 a,b). This finding was comparable to a previous study which showed a similar level of agreement between radiologists and non-radiological physicians and medical students when reading and comprehending radiology reports [ 26 ].…”

“…Matthew’s correlation coefficient (MCC) [ 20 ] was used to compare annotator performance to “gold standard” labeling and to compare annotators’ performance to each other. The MCC was based on values selected for a 2 × 2 confusion matrix ( Table 1 ) where true positive (TP) described the number of labels that matched “gold standard” labels for all positive and negative findings separately.…”

“…To achieve this, MCC was calculated using Python 3.8.10 ( ) with the Pandas [ 21 ] and Numpy [ 22 ] libraries for each label and then micro-averaged [ 23 ] to give an overall coefficient for all positive and negative labels. MCC ranges between −1 and 1, where 1 represents perfect positive correlation, 0 represents correlation not better than random, and −1 represents total disagreement between labels of the “gold standard” set (actual) and the set of labels chosen by the annotator (predicted) [ 20 ].…”

Machine Learning Enables Accurate Prediction of Quinone Formation during Drug Metabolism

“…Matthews Correlation Coefficient is used as a single metric for direct comparison between the developed models. Because MCC is known to be superior to any other performance metric for ranking the binary classification models …”

“…Because MCC is known to be superior to any other performance metric for ranking the binary classification models. 25…”