Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers

Austin, Peter C.; Steyerberg, Ewout W.

doi:10.1002/sim.5941

Cited by 267 publications

(251 citation statements)

References 39 publications

Supporting

Mentioning

248

Contrasting

Unclassified

Order By: Relevance

“…26 Optimism-corrected calibration curves (number of bootstrap samples = 500) and the 45-degree reference line represent ideal model fit and were overlaid on the same plot.…”

Section: Methodsmentioning

confidence: 99%

Risk Prediction Tools to Improve Patient Selection for Carotid Endarterectomy Among Patients With Asymptomatic Carotid Stenosis

et al. 2019

Self Cite

View full text Add to dashboard Cite

IMPORTANCE Randomized clinical trials have demonstrated that patients with asymptomatic carotid stenosis are eligible for carotid endarterectomy (CEA) if the 30-day surgical complication rate is less than 3% and the patient’s life expectancy is at least 5 years. OBJECTIVE To develop a risk prediction tool to improve patient selection for CEA among patients with asymptomatic carotid stenosis. DESIGN, SETTING, AND PARTICIPANTS In this cohort study, veterans 65 years and older who received both carotid imaging and CEA in the Veterans Administration between January 1, 2005, and December 31, 2009 (n = 2325) were followed up for 5 years. Data were analyzed from January 2005 to December 2015. A risk prediction tool (the Carotid Mortality Index [CMI]) based on 23 candidate variables identified in the literature was developed using Veterans Administration and Medicare data. A simpler model based on the number of 4 key comorbidities that were prevalent and strongly associated with 5-year mortality was also developed (any cancer in the past 5 years, chronic obstructive pulmonary disease, congestive heart failure, and chronic kidney disease [the 4C model]). Model performance was assessed using measures of discrimination (eg, area under the curve [AUC])and calibration. Internal validation was performed by correcting for optimism using 500 bootstrapped samples. MAIN OUTCOME AND MEASURE Five-year mortality. RESULTS Among 2325 veterans, the mean (SD) age was 73.74 (5.92) years. The cohort was predominantly male (98.8%) and of white race/ethnicity (94.4%). Overall, 29.5% (n = 687) of patients died within 5 years of CEA. On the basis of a backward selection algorithm, 9 patient characteristics were selected (age, chronic kidney disease, diabetes, chronic obstructive pulmonary disease, any cancer diagnosis in the past 5 years, congestive heart failure, atrial fibrillation, remote stroke or transient ischemic attack, and body mass index) for the final logistic model, which yielded an optimism-corrected AUC of 0.687 for the CMI. The 4C model had slightly worse discrimination (AUC, 0.657) compared with the CMI model; however, the calibration curve was similar to the full model in most of the range of predicted probabilities. CONCLUSIONS AND RELEVANCE According to results of this study, use of the CMI or the simpler 4C model may improve patient selection for CEA among patients with asymptomatic carotid stenosis.

show abstract

“…26 Optimism-corrected calibration curves (number of bootstrap samples = 500) and the 45-degree reference line represent ideal model fit and were overlaid on the same plot.…”

Section: Methodsmentioning

confidence: 99%

Risk Prediction Tools to Improve Patient Selection for Carotid Endarterectomy Among Patients With Asymptomatic Carotid Stenosis

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…As mentioned above, this is not entirely correct, because according to the predictive model that we are externally validating, the sample size for this validation should be independent [9]. However, we must bear in mind that the algorithm we have developed is for scoring systems, which are a specific case of binary logistic regression models.…”

Section: Discussionmentioning

confidence: 99%

Sample size calculation to externally validate scoring systems based on logistic regression models

et al. 2017

View full text Add to dashboard Cite

BackgroundA sample size containing at least 100 events and 100 non-events has been suggested to validate a predictive model, regardless of the model being validated and that certain factors can influence calibration of the predictive model (discrimination, parameterization and incidence). Scoring systems based on binary logistic regression models are a specific type of predictive model.ObjectiveThe aim of this study was to develop an algorithm to determine the sample size for validating a scoring system based on a binary logistic regression model and to apply it to a case study.MethodsThe algorithm was based on bootstrap samples in which the area under the ROC curve, the observed event probabilities through smooth curves, and a measure to determine the lack of calibration (estimated calibration index) were calculated. To illustrate its use for interested researchers, the algorithm was applied to a scoring system, based on a binary logistic regression model, to determine mortality in intensive care units.ResultsIn the case study provided, the algorithm obtained a sample size with 69 events, which is lower than the value suggested in the literature.ConclusionAn algorithm is provided for finding the appropriate sample size to validate scoring systems based on binary logistic regression models. This could be applied to determine the sample size in other similar cases.

show abstract

“…We used two methods for assessing model calibration. First, loess smoothers were used to describe graphically the agreement between predicted probabilities and the observed probabilities of the occurrence of the outcome [9]. Second, we used calibration intercepts and slopes as summary measures [10].…”

Section: Methodsmentioning

confidence: 99%

Geographic and temporal validity of prediction models: different approaches were useful to examine model performance

Austin

Klaveren

Vergouwe

et al. 2016

Journal of Clinical Epidemiology

Self Cite

View full text Add to dashboard Cite

Objective Validation of clinical prediction models traditionally refers to the assessment of model performance in new patients. We studied different approaches to geographic and temporal validation in the setting of multicenter data from two time periods. Study Design and Setting We illustrated different analytic methods for validation using a sample of 14,857 patients hospitalized with heart failure at 90 hospitals in two distinct time periods. Bootstrap resampling was used to assess internal validity. Meta-analytic methods were used to assess geographic transportability. Each hospital was used once as a validation sample, with the remaining hospitals used for model derivation. Hospital-specific estimates of discrimination (c-statistic) and calibration (calibration intercepts and slopes) were pooled using random effects meta-analysis methods. I2 statistics and prediction interval width quantified geographic transportability. Temporal transportability was assessed using patients from the earlier period for model derivation and patients from the later period for model validation. Results Estimates of reproducibility, pooled hospital-specific performance, and temporal transportability were on average very similar, with c-statistics of 0.75. Between-hospital variation was moderate according to I2 statistics and prediction intervals for c-statistics. Conclusion This study illustrates how performance of prediction models can be assessed in settings with multicenter data at different time periods.

show abstract

Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers

Cited by 267 publications

References 39 publications

Risk Prediction Tools to Improve Patient Selection for Carotid Endarterectomy Among Patients With Asymptomatic Carotid Stenosis

Risk Prediction Tools to Improve Patient Selection for Carotid Endarterectomy Among Patients With Asymptomatic Carotid Stenosis

Sample size calculation to externally validate scoring systems based on logistic regression models

Geographic and temporal validity of prediction models: different approaches were useful to examine model performance

Contact Info

Product

Resources

About