A new framework to enhance the interpretation of external validation studies of clinical prediction models

Debray, Thomas P A; Vergouwe, Yvonne; Koffijberg, Hendrik; Nieboer, Daan; Steyerberg, Ewout W.; Moons, Karel G.M.

doi:10.1016/j.jclinepi.2014.06.018

Cited by 401 publications

(501 citation statements)

References 55 publications

Supporting

Mentioning

494

Contrasting

Unclassified

Order By: Relevance

“…Background treatments: Consider differences in treatment use between the development and validation cohorts when exploring the impact of case-mix on model performance [24][25][26].…”

Section: Model Validationmentioning

confidence: 99%

Treatment use in prognostic model research: a systematic review of cardiovascular prognostic studies

et al. 2017

View full text Add to dashboard Cite

Background: Ignoring treatments in prognostic model development or validation can affect the accuracy and transportability of models. We aim to quantify the extent to which the effects of treatment have been addressed in existing prognostic model research and provide recommendations for the handling and reporting of treatment use in future studies. Methods: We first describe how and when the use of treatments by individuals in a prognostic study can influence the development or validation of a prognostic model. We subsequently conducted a systematic review of the handling and reporting of treatment use in prognostic model studies in cardiovascular medicine. Data on treatment use (e.g. medications, surgeries, lifestyle interventions), the timing of their use, and the handling of such treatment use in the analyses were extracted and summarised. Results: Three hundred two articles were included in the review. Treatment use was not mentioned in 91 (30%) articles. One hundred forty-six (48%) reported specific information about treatment use in their studies; 78 (26%) provided information about multiple treatments. Three articles (1%) reported changes in medication use ("treatment drop-in") during follow-up. Seventy-nine articles (26%) excluded treated individuals from their analysis, 80 articles (26%) modelled treatment as an outcome, and of the 155 articles that developed a model, 86 (55%) modelled treatment use, almost exclusively at baseline, as a predictor. Conclusions: The use of treatments has been partly considered by the majority of CVD prognostic model studies. Detailed accounts including, for example, information on treatment drop-in were rare. Where relevant, the use of treatments should be considered in the analysis of prognostic model studies, particularly when a prognostic model is designed to guide the use of certain treatments and these treatments have been used by the study participants. Future prognostic model studies should clearly report the use of treatments by study participants and consider the potential impact of treatment use on the study findings.

show abstract

“…Background treatments: Consider differences in treatment use between the development and validation cohorts when exploring the impact of case-mix on model performance [24][25][26].…”

Section: Model Validationmentioning

confidence: 99%

Treatment use in prognostic model research: a systematic review of cardiovascular prognostic studies

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Recently, a method was proposed to investigate the relatedness of development and validation samples [17]. It uses a model including the envisioned predictors and disease status as covariables to predict membership of an underlying source population for individuals in the derivation and validation samples.…”

Section: Discussionmentioning

confidence: 99%

“…Higher variability of the LP indicates more heterogeneity of case-mix, which implies that individuals have a larger variety of characteristics, suggesting a higher AUC value [17]. For Approach I, we also reported the mean and SD of predictor values among cases and controls to observe the extent to which the two distributions were separated from each other.…”

Section: Analysesmentioning

confidence: 99%

“…With an external validation procedure, one can determine whether the model suffered from 'optimization bias' by comparing its performance in the derivation and validation dataset. When case-mix differences are pronounced, external validation studies examine generalizability [17]. Demonstration of generalizability is more valuable, because it increases the likelihood that the prediction model will also perform well in new subjects.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Impact of Correlation of Predictors on Discrimination of Risk Models in Development and External Populations

Kundu¹,

Mazumdar²,

Ferket³

2016

J Biom Biostat

View full text Add to dashboard Cite

Background: The area under the ROC curve (AUC) of risk models is known to be influenced by differences in case-mix and effect size of predictors. The impact of heterogeneity in correlation among predictors has however been under investigated. We sought to evaluate how correlation among predictors affects the AUC in development and external populations. Methods: We simulated hypothetical populations using two different methods based on means, standard deviations, and correlation of two continuous predictors. In the first approach, the distribution and correlation of predictors were assumed for the total population. In the second approach, these parameters were modeled conditional on disease status. In both approaches, multivariable logistic regression models were fitted to predict disease risk in individuals. Each risk model developed in a population was validated in the remaining populations to investigate external validity. Results: For both approaches, we observed that the magnitude of the AUC in the development and external populations depends on the correlation among predictors. Lower AUCs were estimated in scenarios of both strong positive and negative correlation, depending on the direction of predictor effects and the simulation method. However, when adjusted effect sizes of predictors were specified in the opposite directions, increasingly negative correlation consistently improved the AUC. AUCs in external validation populations were higher or lower than in the derivation cohort, even in the presence of similar predictor effects. Conclusions: Discrimination of risk prediction models should be assessed in various external populations with different correlation structures to make better inferences about model generalizability.

show abstract

“…The new setting in which the model is implemented may be different from the setting in which the prediction model was derived or validated [5,6,25]. The local practices may be different in terms of both medical care and patient populations.…”

Section: Model Performance In the New Settingmentioning

confidence: 99%

Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

et al. 2018

Self Cite

View full text Add to dashboard Cite

An important aim of clinical prediction models is to positively impact clinical decision making and subsequent patient outcomes. The impact on clinical decision making and patient outcome can be quantified in prospective comparative-ideally cluster-randomized-studies, known as 'impact studies'. However, such impact studies often require a lot of time and resources, especially when they are (cluster-)randomized studies. Before envisioning such large-scale randomized impact study, it is important to ensure a reasonable chance that the use of the prediction model by the targeted healthcare professionals and patients will indeed have a positive effect on both decision making and subsequent outcomes. We recently performed two differently designed, prospective impact studies on a clinical prediction model to be used in surgical patients. Both studies taught us new valuable lessons on several aspects of prediction model impact studies, and which considerations may guide researchers in their decision to conduct a prospective comparative impact study. We provide considerations on how to prepare a prediction model for implementation in practice, how to present the model predictions, and how to choose the proper design for a prediction model impact study.

show abstract

A new framework to enhance the interpretation of external validation studies of clinical prediction models

Abstract: The proposed framework enhances the interpretation of findings at external validation of prediction models.

Cited by 401 publications

References 55 publications

Treatment use in prognostic model research: a systematic review of cardiovascular prognostic studies

Treatment use in prognostic model research: a systematic review of cardiovascular prognostic studies

Impact of Correlation of Predictors on Discrimination of Risk Models in Development and External Populations

Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

Contact Info

Product

Resources

About