Updating methods improved the performance of a clinical prediction model in new patients

Janssen, Kristel J.M.; Moons, Karel G.M.; Kalkman, Cor J.; Grobbee, Diederick E.; Vergouwe, Yvonne

doi:10.1016/j.jclinepi.2007.04.018

Cited by 312 publications

(353 citation statements)

References 24 publications

Supporting

Mentioning

337

Contrasting

Unclassified

Order By: Relevance

“…A common test of calibration is the Hosmer-Lemeshow test, which is based on a χ 2 statistic. To cope with poor calibration, several methods for updating models have been suggested [16]. Updating methods range from simple recalibration methods to more sophisticated revision methods.…”

Section: Calibrationmentioning

confidence: 99%

See 1 more Smart Citation

The potential of novel biomarkers to improve risk prediction of type 2 diabetes

et al. 2013

View full text Add to dashboard Cite

The incidence of type 2 diabetes can be reduced substantially by implementing preventive measures in highrisk individuals, but this requires prior knowledge of disease risk in the individual. Various diabetes risk models have been designed, and these have all included a similar combination of factors, such as age, sex, obesity, hypertension, lifestyle factors, family history of diabetes and metabolic traits. The accuracy of prediction models is often assessed by the area under the receiver operating characteristic curve (AROC) as a measure of discrimination, but AROCs should be complemented by measures of calibration and reclassification to estimate the incremental value of novel biomarkers. This review discusses the potential of novel biomarkers to improve model accuracy. The range of molecules that serve as potential predictors of type 2 diabetes includes genetic variants, RNA transcripts, peptides and proteins, lipids and small metabolites. Some of these biomarkers lead to a statistically significant increase of model accuracy, but their incremental value currently seems too small for routine clinical use. However, only a fraction of potentially relevant biomarkers have been assessed with regard to their predictive value. Moreover, serial measurements of biomarkers may help determine individual risk. In conclusion, current risk models provide valuable tools of risk estimation, but perform suboptimally in the prediction of individual diabetes risk. Novel biomarkers still fail to have a clinically applicable impact. However, more efficient use of biomarker data and technological advances in their measurement in clinical settings may allow the development of more accurate predictive models in the future.

show abstract

Section: Calibrationmentioning

confidence: 99%

“…In its simplest form, recalibration means adjusting the intercept of the prediction model leaving the regression coefficients unchanged. A further recalibration method includes adjustment of the intercept and multiplication of all regression coefficients with the same factor (calibration slope) [16].…”

Section: Calibrationmentioning

confidence: 99%

The potential of novel biomarkers to improve risk prediction of type 2 diabetes

et al. 2013

View full text Add to dashboard Cite

show abstract

“…To improve the calibration, the original CART model was updated following standardized procedures. These included updating or adjusting the intercept, followed by more extensive updates if necessary, including model revision (20,29). Initially, the simple recalibration method was used.…”

Section: Data Collection and Definitions Of Candidate Predictors For Esmentioning

confidence: 99%

“…This is known as a difference in case-mix. However, this difference in case-mix, may result in the poor calibration of the original model in the validation population (20,29). As demonstrated in Fig.…”

Section: Cart Model -------------------------------------------------mentioning

confidence: 99%

“…This type of miscalibration, with a slope ~1 and a non-zero intercept, is a typical result in external validation studies, as described by Steyerberg (20). This result indicates that certain patient characteristics (such as age, APACHE-II score and prevalence of disease), which were not included in the prediction model, were distributed differently between the derivation and validation set (20,29). In such cases, a simple recalibration method is often sufficient and preferable (20,29).…”

Section: Cart Model -------------------------------------------------mentioning

confidence: 99%

See 1 more Smart Citation

Extubation outcome after a successful spontaneous breathing trial: A multicenter validation of a 3-factor prediction model

Yang

et al. 2015

Experimental and Therapeutic Medicine

View full text Add to dashboard Cite

Abstract. The aim of the present study was to validate, and if necessary update, a predictive model previously developed using a classification and regression tree (CART) algorithm for predicting successful extubation (ES) using a new cohort. This prospective cohort study enrolled adults admitted to 10 intensive care units, who had successfully passed a spontaneous breathing trial (SBT) and were considered ready for extubation. After extubation, the patients were followed up for 48 h. The primary outcome measure was ES, defined as the ability to maintain spontaneous unassisted breathing for >48 h after extubation. The 3-factor CART model was applied to patients in this cohort. The predicted probability of ES for each patient in this validation cohort was calculated based on the original CART model using the Laplace correction method. The performance was assessed by discrimination and calibration. A decision curve analysis was used assess the clinical net benefit (NB). Extubation failure (EF) occurred in 90/530 patients (17%). Among the 90 patients, 72 (13.6%) were reintubated, while 18 patients remained on rescue noninvasive ventilation within 48 h after extubation. The original CART model showed high discrimination but only moderate calibration with predicted probabilities that were systematically lower than expected. The original CART model was updated, and the updated model preserved excellent discrimination (area under the receiver operating characteristic curve, 0.91; 95% confidence interval, 0.87 to 0.93), but exhibited near-perfect calibration (calibration slope, 1; intercept, 0). Between threshold probabilities of 50 and 80%, the NB of using this updated model is significantly improved compared with the current strategy. The updated CART model may be used to estimate the predicted probability of ES after a successful SBT for individual patients. Applying this model appears to produce a substantial clinical consequence with regard to potential reduction in unexpected EFs.

show abstract

Validation and Utility Testing of Clinical Prediction Models

Adibi

Sadatsafavi

Ioannidis

2020

JAMA

View full text Add to dashboard Cite

The growth in the publication of clinical prediction models (CPMs) has been exponential, largely as a result of an ever-increasingavailabilityofclinicaldata,inexpensivecomputational power, and an expanding tool kit for constructing predictive algorithms. Such an abundance of CPMs has led to an overcrowded, confusing landscape in which it is difficult to identify and select the best, most useful models. 1 Few models are externally validated by the same researchers who developed them, and even fewer by independentinvestigators.Only592(43.3%)of1366cardiovascular CPMs in the Tufts PACE Clinical Predictive Model Registry reported at least 1 validation. 2 The proportions of models in the Tufts registry that reported at least 2, 3, and 10 validations were 20.1%, 12.8%, and 2.9%, respectively. 2 A few select CPMs, such as the Framingham Risk Score and EuroSCORE, have had numerous validations.However, even these models are subject to modifications (eg, adding or removing a predictor variable), with the resulting modified model not revalidated externally. Fragmented efforts that assess only one model at a time do not allow for reliable ranking of the comparative performance of the many CPMs available for the same clinical application. A small number of VIEWPOINT

show abstract

Updating methods improved the performance of a clinical prediction model in new patients

Cited by 312 publications

References 24 publications

The potential of novel biomarkers to improve risk prediction of type 2 diabetes

The potential of novel biomarkers to improve risk prediction of type 2 diabetes

Extubation outcome after a successful spontaneous breathing trial: A multicenter validation of a 3-factor prediction model

Validation and Utility Testing of Clinical Prediction Models

Contact Info

Product

Resources

About