Background
Recent proposals suggest that risk-stratified analyses of clinical trials be routinely performed to better enable tailoring of treatment decisions to individuals. Trial data can be stratified using externally developed risk models (e.g. Framingham risk score), but such models are not always available. We sought to determine whether internally developed risk models, developed directly on trial data, introduce bias compared to external models.
Methods and Results
We simulated a large patient population with known risk factors and outcomes. Clinical trials were then simulated by repeatedly drawing from the patient population assuming a specified relative treatment effect in the experimental arm, which either did or did not vary according to a subjects baseline risk. For each simulated trial, two internal risk models were developed on either the control population only (internal controls only, ICO) or on the whole trial population blinded to treatment (internal whole trial, IWT). Bias was estimated for the internal models by comparing treatment effect predictions to predictions from the external model.
Under all treatment assumptions, internal models introduced only modest bias compared to external models. The magnitude of these biases were slightly smaller for IWT models than for ICO models. IWT models were also slightly less sensitive to bias introduced by overfitting and less sensitive to falsely identifying the existence of variability in treatment effect across the risk spectrum than ICO models.
Conclusions
Appropriately developed internal models produce relatively unbiased estimates of treatment effect across the spectrum of risk. When estimating treatment effect, internally developed risk models using both treatment arms should, in general, be preferred to models developed on the control population.