Developing clinical prediction models when adhering to minimum sample size recommendations: The importance of quantifying bootstrap variability in tuning parameters and predictive performance

Abstract

Recent minimum sample size formula (Riley et al.) for developing clinical prediction models help ensure that development datasets are of sufficient size to minimise overfitting. While these criteria are known to avoid excessive overfitting on average, the extent of variability in overfitting at recommended sample sizes is unknown. We investigated this through a simulation study and empirical example to develop logistic regression clinical prediction models using unpenalised maximum likelihood estimation, and various post-estimation shrinkage or penalisation methods. While the mean calibration slope was close to the ideal value of one for all methods, penalisation further reduced the level of overfitting, on average, compared to unpenalised methods. This came at the cost of higher variability in predictive performance for penalisation methods in external data. We recommend that penalisation methods are used in data that meet, or surpass, minimum sample size requirements to further mitigate overfitting, and that the variability in predictive performance and any tuning parameters should always be examined as part of the model development process, since this provides additional information over average (optimism-adjusted) performance alone. Lower variability would give reassurance that the developed clinical prediction model will perform well in new individuals from the same population as was used for model development.

Development and Internal Validation of a Multivariable Prediction Model to Predict Repeat Attendances in the Pediatric Emergency Department

“…LASSO was used because this performs variable selection (from the list of candidate predictors) through the shrinkage of the model coefficients, while also further helping to minimize overfitting. 26 We evaluated the performance of the model using calibration (agreement between the observed and expected outcomes, across the full risk range) and discrimination (ability of the model to differentiate those who had the outcome from those who did not). Calibration was assessed visually using calibration plots and summarized using the calibration-in-the-large and calibration slope.…”

“…Model hyperparameters were determined by using grid search and selected using 10-fold cross-validation within the training data. LASSO was used because this performs variable selection (from the list of candidate predictors) through the shrinkage of the model coefficients, while also further helping to minimize overfitting 26 …”

External validation of a United Kingdom primary‐care Cushing's prediction tool in a population of referred Dutch dogs

“…A new prediction model was not constructed using the Dutch data set, as the number of cases and non‐cases was relatively low. This could lead to overfitting, rendering the improved or new model not useful for future predictions 35 …”

Stability of clinical prediction models developed using statistical or machine learning methods

“…However, studies have shown that even penalization methods such as LASSO and elastic net are unstable in their selection of predictors (e.g., due to uncertainty in estimation of the tuning parameters from the data, or having a set of highly correlated predictors (Leeuwenberg et al, 2022)), especially in small sample sizes where they are arguably most needed, leading to miscalibration of predictions in new data Van Calster et al, 2020;Van Houwelingen, 2001). For this reason, Martin et al (2021) recommend model developers should use bootstrapping to investigate the uncertainty in their model's penalty terms (shrinkage factors) and predictive performance. Others studies have emphasized using methods to improve stability in the penalization approach (Roberts & Nowak, 2014), such as repeat k-fold cross-validation to estimate penalty or tuning factors from the data Seibold et al, 2018), or ensemble methods that incorporate boosting or bagging (e.g., XGBoost, random forests) (Breiman, 1996).…”