Tufts PACE Clinical Predictive Model Registry: update 1990 through 2015

589

613

When designing a study to develop a new prediction model with binary or time‐to‐event outcomes, researchers should ensure their sample size is adequate in terms of the number of participants (n) and outcome events (E) relative to the number of predictor parameters (p) considered for inclusion. We propose that the minimum values of n and E (and subsequently the minimum number of events per predictor parameter, EPP) should be calculated to meet the following three criteria: (i) small optimism in predictor effect estimates as defined by a global shrinkage factor of ≥0.9, (ii) small absolute difference of ≤ 0.05 in the model's apparent and adjusted Nagelkerke's R2, and (iii) precise estimation of the overall risk in the population. Criteria (i) and (ii) aim to reduce overfitting conditional on a chosen p, and require prespecification of the model's anticipated Cox‐Snell R2, which we show can be obtained from previous studies. The values of n and E that meet all three criteria provides the minimum sample size required for model development. Upon application of our approach, a new diagnostic model for Chagas disease requires an EPP of at least 4.8 and a new prognostic model for recurrent venous thromboembolism requires an EPP of at least 23. This reinforces why rules of thumb (eg, 10 EPP) should be avoided. Researchers might additionally ensure the sample size gives precise estimates of key predictor effects; this is especially important when key categorical predictors have few events in some categories, as this may substantially increase the numbers required.

“…We recommend using

R_{CS_adj}^{2}

Section: How To Prespecify Rboldcs_boldadj2 Based On Previous Informmentioning

confidence: 99%

Minimum sample size for developing a multivariable prediction model: PART II ‐ binary and time‐to‐event outcomes

Riley

Snell

Ensor

et al. 2018

589

613

“…We recommend identifying previous prediction model studies for the same or similar populations and outcomes of interest and extracting their

R_{adj}^{2}

values, which are usually well reported for linear regression models. Helpful for this purpose are systematic reviews of existing models and registries that record the prediction models available in a particular field . If only an

R_{app}^{2}

value is reported in a model development study, then its

R_{adj}^{2}

can be derived using Equation as long as the study's n and p can also be obtained.…”

Section: Sample Size Required To Minimize Overfitting and Optimismmentioning

confidence: 99%

Minimum sample size for developing a multivariable prediction model: Part I – Continuous outcomes

Riley

Snell

Ensor

et al. 2018

173

190

In the medical literature, hundreds of prediction models are being developed to predict health outcomes in individuals. For continuous outcomes, typically a linear regression model is developed to predict an individual's outcome value conditional on values of multiple predictors (covariates). To improve model development and reduce the potential for overfitting, a suitable sample size is required in terms of the number of subjects (n) relative to the number of predictor parameters (p) for potential inclusion. We propose that the minimum value of n should meet the following four key criteria: (i) small optimism in predictor effect estimates as defined by a global shrinkage factor of ≥0.9; (ii) small absolute difference of ≤ 0.05 in the apparent and adjusted R 2 ; (iii) precise estimation (a margin of error ≤ 10% of the true value) of the model's residual standard deviation; and similarly, (iv) precise estimation of the mean predicted outcome value (model intercept). The criteria require prespecification of the user's chosen p and the model's anticipated R 2 as informed by previous studies. The value of n that meets all four criteria provides the minimum sample size required for model development. In an applied example, a new model to predict lung function in African-American women using 25 predictor parameters requires at least 918 subjects to meet all criteria, corresponding to at least 36.7 subjects per predictor parameter. Even larger sample sizes may be needed to additionally ensure precise estimates of key predictor effects, especially when important categorical predictors have low prevalence in certain categories. KEYWORDS continuous outcome, linear regression, minimum sample size, multivariable prediction model, R-squared 1262

“…Common endpoints are the presence of a disease (establishing a diagnosis according to a reference standard) and the occurrence of a future event (prognosis, eg, mortality within 30 days, within 6 months, or longer follow‐up) . Prediction models are increasingly common in the medical literature, and multiple models may be available for the same type of patients for similar endpoints . Published prediction models often use different predictors to derive predictions for individual patients .…”

Section: Introductionmentioning

confidence: 99%

Assessment of heterogeneity in an individual participant data meta‐analysis of prediction models: An overview and illustration

Steyerberg

Nieboer

Debray

et al. 2019

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta‐analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6‐month mortality based on individual patient data using meta‐analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.