Modernising Receiver Operating Characteristic (ROC) Curves

Abstract

The justification for making a measurement can be sought in asking what decisions are based on measurement, such as in assessing the compliance of a quality characteristic of an entity in relation to a specification limit, SL. The relative performance of testing devices and classification algorithms used in assessing compliance is often evaluated using the venerable and ever popular receiver operating characteristic (ROC). However, the ROC tool has potentially all the limitations of classic test theory (CTT) such as the non-linearity, effects of ordinality and confounding task difficulty and instrument ability. These limitations, inherent and often unacknowledged when using the ROC tool, are tackled here for the first time with a modernised approach combining measurement system analysis (MSA) and item response theory (IRT), using data from pregnancy testing as an example. The new method of assessing device ability from separate Rasch IRT regressions for each axis of ROC curves is found to perform significantly better, with correlation coefficients with traditional area-under-curve metrics of at least 0.92 which exceeds that of linearised ROC plots, such as Linacre’s, and is recommended to replace other approaches for device assessment. The resulting improved measurement quality of each ROC curve achieved with this original approach should enable more reliable decision-making in conformity assessment in many scenarios, including machine learning, where its use as a metric for assessing classification algorithms has become almost indispensable.

Detecting Dementia from Face-Related Features with Automated Computational Methods

“…We aim to provide examinable clues to doctors to aid them in making further and detailed analyses of the situation of their patients. Additionally, Pendrill et al [45] revolutionized the conventional ROC curve by incorporating measurement system analysis and the Rasch model. This approach enables the unveiling of crucial insights that traditional metrics fail to capture.…”

Decision Tree Models and Machine Learning Algorithms in the Fault Recognition on Power Lines with Branches

“…However, the use of a statistical approach, as well as methods that are widely used in machine learning problems (for example, [22][23][24][25]) and radio engineering applications (for example, [20]), allow us to introduce the required criteria for evaluating the effectiveness of RPA. It is assumed that the values of the variables n c.t.…”

“…With such a formulation of the problem in the theory of machine learning and radio engineering applications, to assess the recognition ability of trained algorithms [19][20][21][22][23], the "receiver operating characteristic" (ROC-curve) is used [22]. The ROC-curve in relation to the RPA algorithm can be built by plotting on the plane along the ordinate axis the values D, and along the abscissa axis the variable F for various thresholds (settings) of protection operation (Figure 4).…”