The Utility of Receiver Operating Characteristic Curve in Educational Assessment: Performance Prediction

Abstract

When examinees are classified into groups based on scores from educational assessment, two indices are widely used to gauge the psychometric quality of the classifications: accuracy and consistency. The two indices take correct classifications into consideration while overlooking incorrect ones, where unbalanced class distribution threatens the validity of results from the accuracy and consistency indices. The single values produced from the two indices also fail to address the inconsistent accuracy of the classifier across different cut score locations. The current study proposed the concept of classification quality, which utilizes the receiver operating characteristics (ROC) graph to comprehensively evaluate the performance of classifiers. The ROC graph illustrates the tradeoff between benefits (true positive rate) and costs (false positive rate) in classification. In this article, a simulation study was conducted to demonstrate how to generate and interpret ROC graphs in educational assessment and the benefits of using ROC graphs to interpret classification quality. The results show that ROC graphs provide an efficient approach to (a) visualize the fluctuating performance of scoring classifiers, (b) address the unbalanced class distribution issue inherent in the accuracy and consistency indices, and (c) produce more accurate estimation of the classification results.

AAL and Internet of Medical Things for Monitoring Type-2 Diabetic Patients

“…Other commonly used performance assessment indicators, such as the receiver operating characteristic curve (ROC curve) [ 42 ], assess the performance of the classification model. ROC curves consider both the true positive and the false positive measures.…”

Machine learning's model-agnostic interpretability on The Prediction of Students' Academic Performance in Video-Conference-Assisted Online Learning During the Covid-19 Pandemic

“…If there is a possibility of class imbalance in a multi-class classi cation system, micro-average is better. When the classes are imbalanced, the micro average gives more weight to the majority class [31]. We discovered that our dataset was imbalanced since most of the data belonged to class 3 (see Fig.…”

Predictions of success in organic chemistry based on a mathematics skills test and academic achievement

“…The receiver operating characteristic curve offers another indicator of the quality of a classifier such as a logistic regression by plotting sensitivity (or the true positive rates) versus 1specificity (or the false positive rates) at varying cutoffs for the classifier (Fawcett, 2006;Garcı ´a-Valca ´rcel and Tejedor, 2012;Han, 2022). The area under the curve (AUC) indicates the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance; values range from 0 to 1, with a value of 0.5 corresponding to random guessing (Fawcett, 2006).…”

“…The AUC for the logistic regression based on the MUST is 0.840. An AUC is considered acceptable between 0.7 and 0.8, excellent between 0.8 and 0.9, and outstanding between 0.9 and 1.0 (Han, 2022).…”