Individual participant data meta‐analysis for external validation, recalibration, and updating of a flexible parametric prognostic model

Abstract

Individual participant data (IPD) from multiple sources allows external validation of a prognostic model across multiple populations. Often this reveals poor calibration, potentially causing poor predictive performance in some populations. However, rather than discarding the model outright, it may be possible to modify the model to improve performance using recalibration techniques. We use IPD meta‐analysis to identify the simplest method to achieve good model performance. We examine four options for recalibrating an existing time‐to‐event model across multiple populations: (i) shifting the baseline hazard by a constant, (ii) re‐estimating the shape of the baseline hazard, (iii) adjusting the prognostic index as a whole, and (iv) adjusting individual predictor effects. For each strategy, IPD meta‐analysis examines (heterogeneity in) model performance across populations. Additionally, the probability of achieving good performance in a new population can be calculated allowing ranking of recalibration methods. In an applied example, IPD meta‐analysis reveals that the existing model had poor calibration in some populations, and large heterogeneity across populations. However, re‐estimation of the intercept substantially improved the expected calibration in new populations, and reduced between‐population heterogeneity. Comparing recalibration strategies showed that re‐estimating both the magnitude and shape of the baseline hazard gave the highest predicted probability of good performance in a new population. In conclusion, IPD meta‐analysis allows a prognostic model to be externally validated in multiple settings, and enables recalibration strategies to be compared and ranked to decide on the least aggressive recalibration strategy to achieve acceptable external model performance without discarding existing model information.

Applying models of care for total hip and knee arthroplasty: External validation of a published predictive model to identify extended stay risk prior to lower-limb arthroplasty

“…External validation metrics for the original model equation were stored (Supplementary file B) . The model equation was updated as per [24,26] to take into account a corrected intercept and to shrink the model coefficients to account for model optimism during development. The external validation functions were then rerun and the outputs reported.…”

Diabetic retinopathy progression in patients under monitoring for treatment or vision loss: external validation and update of a multivariable prediction model

“…If necessary, we will recalibrate the model to the HES population (eg, by updating the baseline survival function or recalibrating the linear predictor). 27 In addition, we will investigate whether updating the model to include additional predictors that were not available in the development dataset improves the predictive performance. Visual acuity, early worsening, pregnancy and frequent ‘did not attend’, were identified as candidate predictors based on expert opinion and evidence evaluation.…”

Development and Validation of a Multivariable Risk Prediction Model for Sudden Cardiac Death after Myocardial Infarction (PROFID Risk Model): Study Rationale, Design and Protocol

“…As in phase 1, we will apply meta-analysis to pool the estimates of predictive performance of the final aggregate model [46]; this pooled set of performance estimates become our final external validation measures, and we will also calculate prediction intervals for each performance estimate (i.e. the potential model performance in a new population similar to those included in the meta-analysis [47]).…”