Comparison of dynamic updating strategies for clinical prediction models

Abstract

Background Prediction models inform many medical decisions, but their performance often deteriorates over time. Several discrete-time update strategies have been proposed in the literature, including model recalibration and revision. However, these strategies have not been compared in the dynamic updating setting. Methods We used post-lung transplant survival data during 2010-2015 and compared the Brier Score (BS), discrimination, and calibration of the following update strategies: (1) never update, (2) update using the closed testing procedure proposed in the literature, (3) always recalibrate the intercept, (4) always recalibrate the intercept and slope, and (5) always refit/revise the model. In each case, we explored update intervals of every 1, 2, 4, and 8 quarters. We also examined how the performance of the update strategies changed as the amount of old data included in the update (i.e., sliding window length) increased. Results All methods of updating the model led to meaningful improvement in BS relative to never updating. More frequent updating yielded better BS, discrimination, and calibration, regardless of update strategy. Recalibration strategies led to more consistent improvements and less variability over time compared to the other updating strategies. Using longer sliding windows did not substantially impact the recalibration strategies, but did improve the discrimination and calibration of the closed testing procedure and model revision strategies. Conclusions Model updating leads to improved BS, with more frequent updating performing better than less frequent updating. Model recalibration strategies appeared to be the least sensitive to the update interval and sliding window length.

Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

“…We selected the update method by finding the optimal integrated calibration index with minimal impact to discriminative performance. Most of the updating methods exhibited similar performance and all were superior to not updating, similar to other studies that employed model updating for other use cases 11 . In addition, our method included a lag equal to the follow-up period before updating the model to ensure that updating was done prospectively, unlike the approach by Schnellinger et al that was inherently retrospective 11 , which we found to be overly optimistic (see hyperparameter optimization and causal model design in Supplementary Fig.…”

“…There is an inherent tradeoff between stationarity and sample size in the update cohort 51 . While small temporal windows can follow the dynamics of the nonstationary changes in the data, they might not have enough power in samples to enable certain modeling approaches, while larger windows that have adequate sample sizes might not be able to follow quickly changing dynamics 11 . In order to strike the balance for this tradeoff, our approach was to determine the size of the window based on a combination of formulas that estimate the proper sample size for LR models 52 and to perform a hyperparameter optimization that estimates the proper sample size for the custom GLM model (Supplementary Fig.…”

“…These methods could also be implemented as a variation of our sliding window approach, but we found a single updating method with fixed window size sufficient without additional complexity. The closed testing procedure and retraining methods also require more data points than recalibration methods 11 , and reestimation would not have been our optimal choice had fewer patients been initially available resulting in a reduced window size (see hyperparameter optimization in Supplementary Fig. 6).…”

“…These prognostic models are traditionally static 7 . However, model performance can drift and models can lose their predictive abilities over time [7][8][9][10][11][12] . Drifts in the predictive performance of models can appear as a reduction in overall accuracy 13 or miscalibration 8,14,15 and can be due to changes in patient characteristics [14][15][16][17] , as well as new treatments, changes in preventative care, or changes in treatment algorithms 14 .…”

“…Another clinical prognostic model, QRISK, which predicts the risk of a patient developing a heart attack or stroke within ten years is updated annually 15,21,22 . Dynamic updating approaches 7 update the model parameters at a specific time window 11,23,24 or when deemed necessary [10][11][12][13] . For the latter, the frequency of updating depends on the dynamics of the performance drift of such models, making the case for monitoring of prospective model performance.…”