Calculating the sample size required for developing a clinical prediction model

Riley, Richard D; Ensor, Joie; Snell, Kym I E; Harrell, Frank E.; Martin, Glen P.; Reitsma, Johannes B.; Moons, Karel G.M.; Collins, Gary S.; Smeden, Maarten van

doi:10.1136/bmj.m441

Cited by 1,065 publications

(1,079 citation statements)

References 78 publications

(113 reference statements)

Supporting

Mentioning

947

Contrasting

Unclassified

Order By: Relevance

“…Cross-validation is an extension of split-sample validation that uses a larger part of the sample for model development (>80% vs. 50%) (38). While not the most computationally e cient approach, the bootstrap repeated procedure is ideal and expected to produce stable results while conserving the complete study population for validation (22,35,49). In our proposed validation design, a temporal approach to externally validate the model will be explored.…”

Section: Discussionmentioning

confidence: 99%

“…To ensure the development of a robust prediction model for each week gestation from 35 weeks, sample size calculations recommended by Riley et al are provided for stillbirth as a binary outcome to (B1) estimate overall outcome proportion with precision, (B2) target a small mean absolute prediction error, (B3) target a shrinkage factor of 0.9, and (B4) target small optimism of 0.05 in the apparent R 2 (22).…”

Section: Sample Sizementioning

confidence: 99%

See 1 more Smart Citation

Protocol for the development and validation of a risk prediction model for stillbirths from 35 weeks gestation in Australia

Sexton

Coory

Kumar

et al. 2020

Preprint

View full text Add to dashboard Cite

Background: Despite advances in the care of women and their babies in the past century, an estimated 1.7 million babies are born still each year throughout the world. A robust method to estimate a pregnant woman’s individualized risk of late-pregnancy stillbirth may inform decision-making around timing of birth to reduce the risk of stillbirth from 35 weeks gestation in Australia, a high-resource setting.Methods: This is a protocol for a cross-sectional study of all late-pregnancy births in Australia (2005-2015) from 35 weeks gestation including 5,188 stillbirths among 3.1 million births at an estimated rate of 1.7 stillbirths per 1000 births. A multivariable logistic regression model will be developed in line with current Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines to estimate the gestation-specific probability of stillbirth with prediction intervals. Candidate predictors were identified from systematic reviews and clinical consultation and will be assessed through univariable regression analysis. To generate a final model, elimination by backward stepwise multivariable logistic regression will be performed. The model will be internally validated using bootstrapping with 1000 repetitions and externally validated using a temporally unique dataset. Overall model performance will be assessed with R2, calibration, and discrimination. Calibration will be reported using a calibration plot with 95% confidence intervals (α=0.05). Discrimination will be measured by the C-statistic and area underneath the receiver-operator curves. Clinical usefulness will be reported as positive and negative predictive values and a decision curve analysis will be considered. Discussion: A robust method to predict a pregnant woman’s individualized risk of late-pregnancy stillbirth may inform timely, appropriate care to reduce stillbirth. Among existing prediction models designed for maternity use, few have been subject to internal and external validation and many fail to meet recommended reporting standards. In developing a risk prediction model for late-gestation stillbirth with both providers and pregnant women in mind, we endeavor to develop a validated model for clinical use in Australia that meets current reporting standards.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Sample Sizementioning

confidence: 99%

Protocol for the development and validation of a risk prediction model for stillbirths from 35 weeks gestation in Australia

Sexton

Coory

Kumar

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The aim of this study was to evaluate the stability of CVD risk predictions for individual patients when using different sample sizes in the development of the risk prediction models, whilst also considering sample sizes from recent work focusing on over tting and mean absolute prediction error (MAPE), representing state of the art techniques for sample size calculations in risk prediction models. (15)(16)(17) Methods…”

Section: Introductionmentioning

confidence: 99%

Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease

Pate¹,

orcid²,

Emsley³

et al. 2020

Preprint

View full text Add to dashboard Cite

Background: Stability of risk estimates from prediction models may be highly dependent on the sample size of the dataset available for model derivation. In this paper, we evaluate the stability of cardiovascular disease risk scores for individual patients when using different sample sizes for model derivation; such sample sizes include those similar to models recommended in national guidelines, and those based on recently published sample size formula for prediction models.Methods: We mimicked the process of sampling N patients from a population to develop a risk prediction model by sampling patients from the Clinical Practice Research Datalink. A cardiovascular disease risk prediction model was developed on this sample and used to generate risk scores for an independent cohort of patients. This process was repeated 1000 times, giving a distribution of risks for each patient. N = 100 000, 50 000, 10 000, Nmin (derived from sample size formula) and Nepv10 (meets 10 events per predictor rule) were considered. The 5 – 95 percentile range of risks across these models was used to evaluate instability. Patients were grouped by a risk derived from a model developed on the entire population (population derived risk) to summarise results.Results: For a sample size of 100 000, the median 5 – 95 percentile range of risks for patients across the 1000 models was 0.77%, 1.60%, 2.42% and 3.22% for patients with population derived risks of 4-5%, 9-10%, 14-15% and 19-20% respectively, for N = 10 000 it was 2.49%, 5.23%, 7.92% and 10.59%, and for N using the formula derived sample size, it was 6.79%, 14.41%, 21.89% and 29.21%. Restricting this analysis to models with high discrimination, good calibration or small mean absolute prediction error reduced the percentile range, but high levels of instability remained.Conclusions: Widely used cardiovascular disease risk prediction models suffer from high levels of instability induced by sampling variation. Many models will also suffer from overfitting (a closely linked concept), but at acceptable levels of overfitting there may still be high levels of instability in individual risk. Stability of risk estimates should be a criterion when determining the minimum sample size to develop models.

show abstract

“…We will use the term "events per candidate predictor parameter" (EPP) instead, in line with a recent publication. 6 The 10 EPP rule of thumb has shortcomings. 4,6−13 Most importantly, the rule does not guarantee decent risk model performance.…”

Section: Introductionmentioning

confidence: 99%

“…14 Recently, a comprehensive method to determine sample size for prediction model development was proposed, integrating the number of candidate parameters, the assumed event fraction and the anticipated R-squared. 6 This is an important advance, because it requires more detailed argumentation of the anticipated modeling context and it focuses speci cally on prediction model performance.…”

Section: Introductionmentioning

confidence: 99%

Adaptive sample size determination for the development of clinical prediction models

Christodoulou

Smeden

Edlinger³

et al. 2020

Preprint

View full text Add to dashboard Cite

Background: We suggest an adaptive sample size calculation method for developing clinical prediction models, in which model performance is monitored sequentially as new data comes in. Methods: We illustrate the approach using data for the diagnosis of ovarian cancer (n=5914, 33% event fraction) and obstructive coronary artery disease (CAD; n=4888, 44% event fraction). We used logistic regression to develop a prediction model consisting only of a-priori selected predictors and assumed linear relations for continuous predictors. We mimicked prospective patient recruitment by developing the model on 100 randomly selected patients, and we used bootstrapping to internally validate the model. We sequentially added 50 random new patients until we reached a sample size of 3000, and re-estimated model performance at each step. We examined the required sample size for satisfying the following stopping rule: obtaining a calibration slope ≥0.9 and optimism in the c-statistic (ΔAUC) <=0.02 at two consecutive sample sizes. This procedure was repeated 500 times. We also investigated the impact of alternative modeling strategies: modeling nonlinear relations for continuous predictors, and applying Firth’s bias correction.Results: Better discrimination was achieved in the ovarian cancer data (c-statistic 0.9 with 7 predictors) than in the CAD data (c-statistic 0.7 with 11 predictors). Adequate calibration and limited optimism in discrimination was achieved after a median of 450 patients (interquartile range 450-500) for the ovarian cancer data (22 events per parameter (EPP), 20-24), and 750 patients (700-800) for the CAD data (30 EPP, 28-33). A stricter criterion, requiring ΔAUC <=0.01, was met with a median of 500 (23 EPP) and 1350 (54 EPP) patients, respectively. These sample sizes were much higher than the well-known 10 EPP rule of thumb and slightly higher than a recently published fixed sample size calculation method by Riley et al. Higher sample sizes were required when nonlinear relationships were modeled, and lower sample sizes when Firth’s correction was used. Conclusions: Adaptive sample size determination can be a useful supplement to a priori sample size calculations, because it allows to further tailor the sample size to the specific prediction modeling context in a dynamic fashion.

show abstract

Calculating the sample size required for developing a clinical prediction model

Cited by 1,065 publications

References 78 publications

Protocol for the development and validation of a risk prediction model for stillbirths from 35 weeks gestation in Australia

Protocol for the development and validation of a risk prediction model for stillbirths from 35 weeks gestation in Australia

Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease

Adaptive sample size determination for the development of clinical prediction models

Contact Info

Product

Resources

About