Validation and updating of predictive logistic regression models: a study on sample size and shrinkage

Steyerberg, Ewout W.; Borsboom, Gerard; Houwelingen, Hans C. van; Eijkemans, Marinus J.C.; Habbema, J. Dik F.

doi:10.1002/sim.1844

Cited by 460 publications

(532 citation statements)

References 35 publications

(35 reference statements)

Supporting

Mentioning

508

Contrasting

Unclassified

Order By: Relevance

“…Recalibration of existing models is attractive because of the stability which is related to the fact that only two parameters (intercept and calibration slope) are estimated [20] (see Methods section). Directly using the observed risk pertaining to a certain risk class is hampered through the potential imprecision due to the small number of observations within a certain class of predicted risk ( Table 2).…”

Section: Discussionmentioning

confidence: 99%

Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings

et al. 2008

View full text Add to dashboard Cite

SUMMARYIn patients with community-acquired pneumonia (CAP) prediction rules based on individual predicted mortalities are frequently used to support decision-making for in-patient vs. outpatient management. We studied the accuracy and the need for recalibration of three risk prediction scores in a tertiary-care University hospital emergency-department setting in Switzerland. We pooled data from patients with CAP enrolled in two randomized controlled trials. We compared expected mortality from the original pneumonia severity index (PSI), CURB65 and CRB65 scores against observed mortality (calibration) and recalibrated the scores by fitting the intercept a and the calibration slope b from our calibration model. Each of the original models underestimated the observed 30-day mortality of 11 %, in 371 patients admitted to the emergency department with CAP (8 . 4%, 5 . 5 % and 5 . 0% for the PSI, CURB65 and CRB65 scores, respectively). In particular, we observed a relevant mortality within the low risk classes of the original models (2 . 6%, 5 . 3%, and 3 . 7% for PSI classes I-III, CURB65 classes 0-1, and CRB65 class 0, respectively). Recalibration of the original risk models corrected the miscalibration. After recalibration, however, only PSI class I was sensitive enough to identify patients with a low risk (i.e. <1%) for mortality suitable for outpatient management. In our tertiary-care setting with mostly referred in-patients, CAP risk scores substantially underestimated observed mortalities misclassifying patients with relevant risks of death suitable for outpatient management. Prior to the implementation of CAP risk scores in the clinical setting, the need for recalibration and the accuracy of low-risk re-classification should be studied in order to adhere with discharge guidelines and guarantee patients' safety.

show abstract

Section: Discussionmentioning

confidence: 99%

Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings

et al. 2008

View full text Add to dashboard Cite

show abstract

“…An adjusted risk score is the risk score obtained by fitting the basic risk score to the validation sample using an intercept and slope (Cox, 1958;Steyerberg et al 2004). One form of the adjusted risk score is…”

Section: Risk Scoresmentioning

confidence: 99%

Evaluating a New Marker for Risk Prediction Using the Test Tradeoff: An Update

Baker

Calster²

2012

The International Journal of Biostatistics

Self Cite

View full text Add to dashboard Cite

Most of the methodological literature on evaluating an additional marker for risk prediction involves purely statistical measures of classification performance. A disadvantage of a purely statistical measure is the difficulty in deciding the improvement in the measure that would make inclusion of the additional marker worthwhile. In contrast, a medical decision making approach can weigh the cost or harm of ascertaining an additional marker against the benefit of a higher true positive rate for a given false positive rate that may be associated with risk prediction involving the additional marker. An appealing form of the medical decision making approach involves the risk threshold, which is the risk at which the expected utility of treatment and no treatment is the same. In this framework, a readily interpretable evaluation of the net benefit of an additional marker is the test tradeoff corresponding to the risk threshold. The test tradeoff is the minimum number of tests for a new marker that need to be traded for a true positive to yield an increase in the net benefit of risk prediction with the additional marker. For a sensitivity analysis the test tradeoff is computed over multiple risk thresholds. This article updates the theory and estimation of the test tradeoff. An example is provided.

show abstract

“…Nevertheless, we recommend a process of continuous updating of the models, and furthermore, extensive assessment of their validity in other settings. 28 The explorations for improving the design and analysis of clinical trials in TBI illustrate how crude our approaches over the past decades have really been and that we may not have been giving new therapies a fair chance. The recommendation for relatively broad inclusion criteria, performing prespecified covariate adjustment in the analysis phase, constitutes a clear change of direction from current approaches in which studies tend to use strict enrolment criteria in an attempt to decrease the inherent heterogeneity of the population.…”

Section: Discussionmentioning

confidence: 99%

IMPACT Recommendations for Improving the Design and Analysis of Clinical Trials in Moderate to Severe Traumatic Brain Injury

et al. 2010

Self Cite

View full text Add to dashboard Cite

Summary: Clinical trials in traumatic brain injury (TBI) pose complex methodological challenges, largely related to the heterogeneity of the population. The International Mission on Prognosis and Clinical Trial Design in TBI study group has explored approaches for dealing with this heterogeneity with the aim to optimize clinical trials in TBI. Extensive prognostic analyses and simulation studies were conducted on individual patient data from eight trials and three observational studies. Here, we integrate the results of these studies into the International Mission on Prognosis and Clinical Trial Design in TBI recommendations for design and analysis of trials in TBI:• Details of the major baseline prognostic characteristics should be provided in every report on a TBI study; in trials they should be differentiated per treatment group. We also advocate the reporting of the baseline prognostic risk as determined by validated prognostic models.• Inclusion criteria should be as broad as is compatible with the current understanding of the mechanisms of action of the intervention being evaluated. This will maximize recruitment rates and enhance the generalizability of the results.• The statistical analysis should incorporate prespecified covariate adjustment to mitigate the effects of the heterogeneity.• The statistical analysis should use an ordinal approach, based on either sliding dichotomy or proportional odds methodology.Broad inclusion criteria, prespecified covariate adjustment, and an ordinal analysis will promote an efficient trial, yielding gains in statistical efficiency of more than 40%. This corresponds to being able to detect a 7% treatment effect with the same number of patients needed to demonstrate a 10% difference with an unadjusted analysis based on the dichotomized Glasgow outcome scale.

show abstract

Validation and updating of predictive logistic regression models: a study on sample size and shrinkage

Cited by 460 publications

References 35 publications

Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings

Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings

Evaluating a New Marker for Risk Prediction Using the Test Tradeoff: An Update

IMPACT Recommendations for Improving the Design and Analysis of Clinical Trials in Moderate to Severe Traumatic Brain Injury

Contact Info

Product

Resources

About