Decision curve analysis: a technical note

Abstract

Multivariable regression models are widely used in medical literature for the purpose of diagnosis or prediction. Conventionally, the adequacy of these models is assessed using metrics of diagnostic performances such as sensitivity and specificity, which fail to account for clinical utility of a specific model. Decision curve analysis (DCA) is a widely used method to measure this utility. In this framework, a clinical judgment of the relative value of benefits (treating a true positive case) and harms (treating a false positive case) associated with prediction models is made. As such, the preferences of patients or policy-makers are accounted for by using a metric called threshold probability. A decision analytic measure called net benefit is then calculated for each possible threshold probability, which puts benefits and harms on the same scale. The article is a technical note on how to perform DCA in R environment. The decision curve is depicted with the system. Correction for overfitting is done via either bootstrap or cross-validation. Confidence interval and P values for the comparison of two models are calculated using bootstrap method. Furthermore, we describe a method for computing area under net benefit for the comparison of two models. The average deviation about the probability threshold (ADAPT), which is a more recently developed index to measure the utility of a prediction model, is also introduced in this article.

Nomogram based on inflammatory indices for differentiating intrahepatic cholangiocarcinoma from hepatocellular carcinoma

“…There was also a good calibration curve for the ICC risk estimation (Figure ). DCA has been used to assess the clinical value of models which integrates the preferences of patients into the analysis . DCA showed that using this nomogram to distinguish ICC from HCC added more benefit compared with model 1 and 2 (Figure ).…”

Is it feasible to predict lymph node metastasis intraoperatively or postoperatively in early-stage lung adenocarcinoma: the application of machine learning algorithms?

“…Another methodological problem remained to be further explained is that the false-positive and false-negative rate need to be low enough to achieve good clinical utility. High AUC in ROC represents high predictive accuracy but does not necessary prove good clinical utility, because false-positive or false-negative results could reduce net bene t [36]. To seek for a model that has high predictive accuracy and net bene t, we adopted DCA which has been widely proven to be e ciently and interpretable in the evaluation of clinical utility [37].…”

Development and external validation of an online clinical prediction model for augmented renal clearance in adult mixed critically ill patients: the ARC predictor

“…A prediction model is clinically useful if its net benefit is higher than for the alternative strategies (i.e. 'all ARC' and 'none ARC') (27). As shown in this figure, the ARC predictor shows net benefit above the alternative strategies over a broad range of threshold probabilities (0.01-0.71), both in the internal validation set and in the external validation…”

“…'all ARC' meaning "assume all days show ARC", or 'none ARC' meaning "assume none of the days show ARC") over a range of threshold probabilities that would be used in clinical practice. A threshold probability is the predicted probability above which a patient would be classified as showing ARC on the next day [26][27][28]. Performance of the model was subsequently assessed in the internal validation set.…”