Evaluating disease prediction models using a cohort whose covariate distribution differs from that of the target population

Abstract

Personal predictive models for disease development play important roles in chronic disease prevention. The performance of these models is evaluated by applying them to the baseline covariates of participants in external cohort studies, with model predictions compared to subjects’ subsequent disease incidence. However the covariate distribution among participants in a validation cohort may differ from that of the population for which the model will be used. Since estimates of predictive model performance depend on the distribution of covariates among the subjects to which it is applied, such differences can cause misleading estimates of model performance in the target population. We propose a method for addressing this problem by weighting the cohort subjects to make their covariate distribution better match that of the target population. Simulations show that the method provides accurate estimates of model performance in the target population, while un-weighted estimates may not. We illustrate the method by applying it to evaluate an ovarian cancer prediction model targeted to US women, using cohort data from participants in the California Teachers Study. The methods can be implemented using open-source code for public use as the R-package RMAP (Risk Model Assessment Package) available at http://stanford.edu/~ggong/rmap/.

Calibrating validation samples when accounting for measurement error in intervention studies

“…We show that weighting the validation sample to better resemble the intervention study can help reduce such biases, and improve upon the transportability of measurement error parameters estimated in an external sample. Previous work has demonstrated similar benefits of implementing propensity score-type weighting methods when using external validation data to adjust for missing confounders 36 and when evaluating disease prediction models in samples that differ from the target population 37 Through our simulation study, we highlighted that under extreme scenarios, it may still be inappropriate to transport if the validation sample is vastly different from the trial on a set of observed characteristics. It is also important to remember that while researchers are often concerned about measurement error, it will only lead to a biased ATE estimate when the outcome error is differential with respect to treatment and the error means are thereby different across treatment groups.…”

Modeling risks of cardiovascular and cancer mortality following a diagnosis of loco-regional breast cancer

“…In this approach, patients who experienced an event after the prediction time frame were considered non-events. The second approach we applied used all patients, and those who did not have complete follow-up for the prediction horizon contributed data only up to the length of time that they were followed, such that competing events are not excluded from the validation sample, but rather censored at their last follow-up time [ 28 ]. We used the rmap package in R for this latter approach.…”

Towards a modular decision support system for radiomics: A case study on rectal cancer

“…Similarly, features can be analysed in order to identify those which are more robust to common sources of image noise. However, despite the fact that empirical investigations which rely on external validation are deemed to be qualitative better than others by the TRIPOD guidelines [19], this approach can not guarantee the reproducibility of the observed results in every set [20].…”