Minimum sample size for developing a multivariable prediction model: PART II ‐ binary and time‐to‐event outcomes

Abstract

When designing a study to develop a new prediction model with binary or time‐to‐event outcomes, researchers should ensure their sample size is adequate in terms of the number of participants (n) and outcome events (E) relative to the number of predictor parameters (p) considered for inclusion. We propose that the minimum values of n and E (and subsequently the minimum number of events per predictor parameter, EPP) should be calculated to meet the following three criteria: (i) small optimism in predictor effect estimates as defined by a global shrinkage factor of ≥0.9, (ii) small absolute difference of ≤ 0.05 in the model's apparent and adjusted Nagelkerke's R2, and (iii) precise estimation of the overall risk in the population. Criteria (i) and (ii) aim to reduce overfitting conditional on a chosen p, and require prespecification of the model's anticipated Cox‐Snell R2, which we show can be obtained from previous studies. The values of n and E that meet all three criteria provides the minimum sample size required for model development. Upon application of our approach, a new diagnostic model for Chagas disease requires an EPP of at least 4.8 and a new prognostic model for recurrent venous thromboembolism requires an EPP of at least 23. This reinforces why rules of thumb (eg, 10 EPP) should be avoided. Researchers might additionally ensure the sample size gives precise estimates of key predictor effects; this is especially important when key categorical predictors have few events in some categories, as this may substantially increase the numbers required.

No association between perceived exertion and session duration with hamstring injury occurrence in professional football

“…Since this is the first study to examine the relationship between load and hamstring injury in football, a formal a priori sample size estimation was not possible using existing studies as per the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement 22‐item checklist . Accordingly, to inform the design of future studies, Cox‐Snell pseudo‐ R 2 ( R 2 CS ) statistics were reported as measures of model overall performance . Outcome statistics are reported as point estimates and 95% confidence intervals (CI).…”

“…Given the novelty of our study, a formal a priori sample size estimation informed by the precision of coefficient estimates or relevant model statistics from any existing study could not be performed. Nevertheless, recent advances in the procedures for determining minimum sample size now permit a robust appraisal of the sample size requirements based on pseudo‐ R 2 statistics . Therefore, we reported the recommended statistics, which can be used by researchers and clinicians to inform sample size estimation for future investigations in this field (Table ).…”

“…Nevertheless, recent advances in the procedures for determining minimum sample size now permit a robust appraisal of the sample size requirements based on pseudo‐ R 2 statistics . Therefore, we reported the recommended statistics, which can be used by researchers and clinicians to inform sample size estimation for future investigations in this field (Table ). For example, in the case of the model with the 28‐day cumulative session duration, assuming a population outcome prevalence of 0.309 and using the R 2 CS value of 0.074 in the equation indicate a minimum sample size requirement of 329, 583, 1166 players for the development of new models with one, five, and ten load‐related candidate predictor parameters, respectively.…”

When and how to use data from randomised trials to develop or validate prognostic models

“…While no consensus currently exists on the sample sizes required for prognostic prediction, the required sample size depends on several factors, including the number of predictors, total sample size, and number or proportion of events 5859. Thus, large sample sizes could be needed to reliably develop a prognostic model, especially when tens or hundreds of candidate predictor variables are considered.…”

New Guidelines for Data Reporting and Statistical Analysis: Helping Authors With Transparency and Rigor in Research

“…(17) However, recent studies have shown that such a rule of thumb is not adequate because the number of events is dependent on the number of risk factors/parameters, proportion of variance explained, shrinkage factor (ie, bias between sample statistics and population parameters), and incidence of disease. (18) For example, a study on fracture risk with 10 risk factors, assuming 10% fracture incidence, shrinkage factor of 0.90, and 20% expected variance explained, would require approximately 400 individuals for a multivariable logistic regression model. However, if the proportion of variance explained is 10%, the required sample size would increase to 850 individuals.…”