Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease

Abstract

Background: Stability of risk estimates from prediction models may be highly dependent on the sample size of the dataset available for model derivation. In this paper, we evaluate the stability of cardiovascular disease risk scores for individual patients when using different sample sizes for model derivation; such sample sizes include those similar to models recommended in national guidelines, and those based on recently published sample size formula for prediction models.Methods: We mimicked the process of sampling N patients from a population to develop a risk prediction model by sampling patients from the Clinical Practice Research Datalink. A cardiovascular disease risk prediction model was developed on this sample and used to generate risk scores for an independent cohort of patients. This process was repeated 1000 times, giving a distribution of risks for each patient. N = 100 000, 50 000, 10 000, Nmin (derived from sample size formula) and Nepv10 (meets 10 events per predictor rule) were considered. The 5 – 95 percentile range of risks across these models was used to evaluate instability. Patients were grouped by a risk derived from a model developed on the entire population (population derived risk) to summarise results.Results: For a sample size of 100 000, the median 5 – 95 percentile range of risks for patients across the 1000 models was 0.77%, 1.60%, 2.42% and 3.22% for patients with population derived risks of 4-5%, 9-10%, 14-15% and 19-20% respectively, for N = 10 000 it was 2.49%, 5.23%, 7.92% and 10.59%, and for N using the formula derived sample size, it was 6.79%, 14.41%, 21.89% and 29.21%. Restricting this analysis to models with high discrimination, good calibration or small mean absolute prediction error reduced the percentile range, but high levels of instability remained.Conclusions: Widely used cardiovascular disease risk prediction models suffer from high levels of instability induced by sampling variation. Many models will also suffer from overfitting (a closely linked concept), but at acceptable levels of overfitting there may still be high levels of instability in individual risk. Stability of risk estimates should be a criterion when determining the minimum sample size to develop models.

Uncertainty and the Value of Information in Risk Prediction Modeling

“…The adequacy of development sample of a given size has been investigated in terms of 'stability' of predictions. Pate et al have shown that typical sizes of development samples result in unstable predictions, manifested in large variations in predicted risks for the same patient using random development samples of identical sizes (4). The authors recommended a bootstrap approach for examining the stability of risk predictions.…”

“…2,3 The adequacy of the development sample of a given size has also been investigated in terms of the stability of predictions. 4 Despite targeting different objectives, such approaches are fundamentally concerned with the accuracy of predictions from a purely statistical perspective. Given that risk prediction models are used for patient care, of ultimate relevance is to what extent such uncertainty affects the outcome of treatment decisions.…”

Developing clinical prediction models when adhering to minimum sample size recommendations: The importance of quantifying bootstrap variability in tuning parameters and predictive performance

“…Note, we are interested in variability in overall performance, rather than stability of individual risks as studied recently. 28 The remainder of the paper is structured as follows: section 'Shrinkage and penalisation methods to developing prediction models' describes the common approaches to develop CPMs using penalisation; section 'Riley et al sample size criteria' gives a brief overview of the Riley et al sample size criteria; section 'Simulation study' describes the methods and results of our simulation study; while section 'Empirical study' reports the results from the real-world critical care example. Finally, concluding remarks are given in section the 'Discussion' section.…”