Assessing the Performance of Prediction Models

Abstract

The performance of prediction models can be assessed using a variety of different methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic (ROC) curve), and goodness-of-fit statistics for calibration. Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision–analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions. We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n=544 for model development, n=273 for external validation). We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for making clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.

Prediction models for the risk of spontaneous preterm birth based on maternal characteristics: a systematic review and independent external validation

“…We assessed the predictive performance of each model by means of discrimination and calibration for the outcomes sPTB <37 and <34 weeks of gestation, as described in the framework reported by Steyerberg et al. 16. Discrimination indicates the ability of the model to distinguish between women who will have a sPTB and those who will not.…”

“…Discriminative performance (AUC) of the models is not affected, as this recalibration method does not change the ranking of the predicted probabilities 24. A discriminative performance below 0.70 is generally considered moderate 16.…”

“…In the last decade, a number of promising prediction models based on multivariable regression analysis for the risk of sPTB have been published 15. Prediction models may be more accurate in identifying women at high risk, as regression allows for a more fine‐tuned estimation of the weight of multiple risk factors and possible inter‐relations 16. A review of all existing models assessing their methodological quality is lacking.…”

Validating performance of TRISS, TARN and NORMIT survival prediction models in a Norwegian trauma population

“…AUROC is mathematically equivalent to the c statistic, which denotes the proportion of all possible pairs of patients drawn from the population, one a survivor and one a non‐survivor, where the patient who survived had the higher P s. AUROC with 95% confidence intervals (95% CI) for all models were compared, and non‐overlapping 95% CIs were considered a significant difference in discrimination ability. We calculated the discrimination slope with 95% CI for each model, i.e., the absolute difference between the mean predicted probability of survival ( P s) for survivors and for non‐survivors 24. We also calculated the median P s with 95% CI for survivors and for non‐survivors, as the distributions of P s values were highly skewed.…”

International Validation of the Thrombolysis in Myocardial Infarction (TIMI) Risk Score for Secondary Prevention in Post‐MI Patients: A Collaborative Analysis of the Chronic Kidney Disease Prognosis Consortium and the Risk Validation Scientific Committee

“…Since the issue of calibration may be fixable by recalibration,23 as seen in some of our cohorts, discrimination ability is essential for risk prediction 24. In our study, the ability of TRS2°P to discriminate the risk of subsequent cardiovascular outcomes among patients with MI was reasonably good.…”