Calibration Belt for Quality-of-Care Assessment Based on Dichotomous Outcomes

Finazzi, Stefano; Poole, Daniele; Luciani, Davide; Cogo, Paola; Bertolini, Guido

doi:10.1371/journal.pone.0016110

Cited by 132 publications

(135 citation statements)

References 13 publications

Supporting

Mentioning

135

Contrasting

Order By: Relevance

“…Why not use both the traditional approach and the new one and compare them? We must accept that the mathematical basis of the new approach has been shown elsewhere [9]. On such theoretical grounds, the internal validity is possibly sufficient to grant external validity.…”

mentioning

confidence: 93%

“…The traditional plot or calibration curve also has some disadvantages: first, rather than a curve, it is a jagged line connecting the points in the plot; second, it is not accompanied by any information on the statistical significance of deviations from the bisector [9].…”

mentioning

confidence: 99%

“…Discrimination was measured as usual using ROC curves and both models discriminated fairly. Interestingly, the calibration ability of the systems was measured using a new approach named calibration belt [9].…”

mentioning

confidence: 99%

“…Although Poole et al focused in this study on the comparison between SAPS II and SAPS 3, in our opinion the main relevance of their work is to introduce the implementation of the calibration belt approach [9] as a new calibration-assessment tool. Calibration of SAPS II and SAPS 3 mortality predictions was measured using the new method.…”

mentioning

confidence: 99%

“…This method also enables confidence intervals to be computed for the curve, which can be plotted as a calibration belt. This approach allows one to discriminate the ranges in which the model miscalibrates in addition to indicating the direction of this phenomenon [9]. We can argue that the H-L statistics associated with the traditional calibration plot would have provided more or less the same information.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Calibration strategies to validate predictive models: is new always better?

Serrano

2012

Intensive Care Med

View full text Add to dashboard Cite

Calibration along with discrimination is an important measure of accuracy to validate predictive logistic regression models. Most predictive models in intensive care such as Simplified Acute Physiology Score (SAPS) II [1] and SAPS 3 [2,3] consider the binary outcome whether a patient will be alive or dead at hospital discharge. Discrimination measures how well the model can distinguish between patients who die and those who survive. Discrimination is usually assessed by the area under the receiver operating characteristic curve (AU-ROC) [4]. This statistic evaluates each pair of observations that have different outcomes and calculates the proportion of times when the patient who died had a higher predicted mortality than did the survivor. The AU-ROC ranges from 0.50 (no discrimination: complete binary random of 50 % similar to flipping a coin) to 1.00 (100 % correct discrimination of the model) [4].Calibration measures the model's ability to generate predictions that are on average close to the average observed outcome. Calibration has traditionally been approached in two steps. First, to investigate the overall ability to correctly relate the actual occurrence of the event to its estimated probability using statistical methods. The most widely used method is the HosmerLemeshow (H-L) test [5], which examines how well the percentage of observed deaths matches the percentage of predicted deaths over deciles of predicted risk. A p value greater than 0.05 is needed to conclude that there are no significant differences between the observed and expected outcomes and therefore the model has good overall calibration. Second, to localize possible deviations across risk strata by means of calibration plots with observed outcomes versus expected probabilities of mortality. The calibration plot, also named calibration curve, intends to provide complementary information over subsets. If the model calibrates well, there will not be a substantial deviation from the 45°line of perfect fit or bisector. On the contrary, miscalibration of the model will be a function of expected probability.The H-L test is easy to compute and its interpretation is intuitive, but it has acknowledged limitations such as being very sensitive to sample size [6][7][8]. The traditional plot or calibration curve also has some disadvantages: first, rather than a curve, it is a jagged line connecting the points in the plot; second, it is not accompanied by any information on the statistical significance of deviations from the bisector [9].Methods to measure the calibration of predictive models have remained unchanged for a long time, but new insights are now provided. In this issue of Intensive Care Medicine, Poole et al.[10] present a multicenter study aimed at comparing the performance of SAPS II and SAPS 3 in predicting hospital mortality. Discrimination was measured as usual using ROC curves and both models discriminated fairly. Interestingly, the calibration ability of the systems was measured using a new approach named calibration belt [9].Both syste...

show abstract

mentioning

confidence: 93%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Calibration strategies to validate predictive models: is new always better?

Serrano

2012

Intensive Care Med

View full text Add to dashboard Cite

show abstract

Calibration of Prognostic Risk Scores

Calster

2018

Wiley StatsRef: Statistics Reference Online

View full text Add to dashboard Cite

Prognostic risk scores provide individualized risk estimates for an outcome (“event”) based on included covariates. Calibration indicates the extent to which risk estimates are reliable. We review four increasingly stringent definitions of calibration: mean, weak, moderate, and strong calibration. The latter requires that the risk model perfectly corresponds to the observed proportions for every single covariate pattern, and hence can be regarded as impossible to achieve in practice (“utopic”). We present an overview of methods to assess calibration, including goodness‐of‐fit tests, summary measures, and graphical procedures. We show that estimation and visualization of the degree of (mis)calibration is essential, whereas testing is problematic. We illustrate the methods with a case study on the prediction of the histology of retroperitoneal lymph nodes following chemotherapy for testicular cancer.

show abstract

Energy spectrum CT index‐based machine learning model predicts the effect of intravenous thrombolysis in lower limbs

Liu,

Yang,

Yin

et al. 2023

J Applied Clin Med Phys

View full text Add to dashboard Cite

To develop a noninvasive machine learning (ML) model based on energy spectrum computed tomography venography (CTV) indices for preoperatively predicting the effect of intravenous thrombolytic treatment in lower limbs. A total of 3492 slices containing thrombus regions from 58 veins in lower limbs in a cohort of 18 patients, divided in good and poor thrombolysis prognosis groups, were analyzed. Key indices were selected by univariate analysis and Pearson correlation coefficient test. A support vector machine classifier‐based model was developed through ten‐fold cross validation. Model performance was assessed in terms of discrimination, calibration, and clinical usefulness at both per‐slice and per‐vessel levels. Continuous variables and categorical variables were compared between good and poor thrombolysis prognosis group by Mann‐Whitney U‐test and chi‐square test, respectively. A nomogram was built by integrating clinical factors and the energy spectrum CTV index‐based score calculated by the model. Six indices selected from 192 indices were used to build the predictive model. The ML model achieved area under the curves (AUCs) of 0.838 and 0.767 [95% CI (confidence interval), 0.825–0.850, 0.752–0.781] in the training and validation datasets at the per‐slice level, and the per‐vessel level AUCs were 0.945 and 0.876 (95% CI, 0.852–0.988, 0.763–0.948) in the training and validation datasets, respectively. The nomogram showed better performance with the per‐vessel level AUC, accuracy, sensitivity and specificity, yielding 0.901(95% CI, 0.793–0.964), 86.2%, 87.9% and 84.0% in the validation dataset, respectively. There was no significant difference in the vessel distribution between good and poor thrombolysis prognosis groups (chi‐square test, p = 0.671). The energy spectrum CTV index‐based ML model achieved favorable effectiveness in predicting the outcome of vessel‐level intravenous thrombolysis. A nomogram integrating clinical factors, and risk score calculated by the developed model showed improved performance and had potential to be used as a noninvasive preoperative tool for clinicians.

show abstract

Calibration Belt for Quality-of-Care Assessment Based on Dichotomous Outcomes

Cited by 132 publications

References 13 publications

Calibration strategies to validate predictive models: is new always better?

Calibration strategies to validate predictive models: is new always better?

Calibration of Prognostic Risk Scores

Energy spectrum CT index‐based machine learning model predicts the effect of intravenous thrombolysis in lower limbs

Contact Info

Product

Resources

About