Measuring diagnostic and predictive accuracy in disease management: an introduction to receiver operating characteristic (ROC) analysis

Abstract

Diagnostic or predictive accuracy concerns are common in all phases of a disease management (DM) programme, and ultimately play an influential role in the assessment of programme effectiveness. Areas, such as the identification of diseased patients, predictive modelling of future health status and costs and risk stratification, are just a few of the domains in which assessment of accuracy is beneficial, if not critical. The most commonly used analytical model for this purpose is the standard 2 x 2 table method in which sensitivity and specificity are calculated. However, there are several limitations to this approach, including the reliance on a single defined criterion or cut-off for determining a true-positive result, use of non-standardized measurement instruments and sensitivity to outcome prevalence. This paper introduces the receiver operator characteristic (ROC) analysis as a more appropriate and useful technique for assessing diagnostic and predictive accuracy in DM. Its advantages include; testing accuracy across the entire range of scores and thereby not requiring a predetermined cut-off point, easily examined visual and statistical comparisons across tests or scores, and independence from outcome prevalence. Therefore the implementation of ROC as an evaluation tool should be strongly considered in the various phases of a DM programme.

Application of machine learning algorithms for clinical predictive modeling: a data-mining approach in SCT

“…17,18,42,43 Models can also be evaluated according to other measures. 17 Interpretability refers to the level of understanding and insight that is provided.…”

Metabolic profiling of human blood

“…Одним из наиболее удобных методов для оценки диагностической значимости выявляемых биомаркеров является ROC-кривая (Receiver-Operator Characteristic) [27]. ROC-кривая строится по значениям следующих показателей -диагностическая чувствительность, показывающая долю истинно положительных результатов у всех лиц с исследуемой патологией, и диагностическая специфичность, показывающая долю истинно отрицательных результатов у всех лиц не имеющих исследуемой патологии (здоровых) (рис.…”

Comprehensive epidermal growth factor receptor gene analysis from cytological specimens of non-small-cell lung cancers

“…In order to synchronise the FISH status of cytological and histological specimens, we tuned the threshold of a positive FISH result for cytological specimens using ROC curve based on the 33 cases with matched specimens (Linden, 2006). Changing the number of gene copies and the minimal percentage of cells in cytological specimens would identify the ideal threshold with maximal sensitivity and specificity.…”