Using Internally Developed Risk Models to Assess Heterogeneity in Treatment Effects in Clinical Trials

Abstract

Background Recent proposals suggest that risk-stratified analyses of clinical trials be routinely performed to better enable tailoring of treatment decisions to individuals. Trial data can be stratified using externally developed risk models (e.g. Framingham risk score), but such models are not always available. We sought to determine whether internally developed risk models, developed directly on trial data, introduce bias compared to external models. Methods and Results We simulated a large patient population with known risk factors and outcomes. Clinical trials were then simulated by repeatedly drawing from the patient population assuming a specified relative treatment effect in the experimental arm, which either did or did not vary according to a subjects baseline risk. For each simulated trial, two internal risk models were developed on either the control population only (internal controls only, ICO) or on the whole trial population blinded to treatment (internal whole trial, IWT). Bias was estimated for the internal models by comparing treatment effect predictions to predictions from the external model. Under all treatment assumptions, internal models introduced only modest bias compared to external models. The magnitude of these biases were slightly smaller for IWT models than for ICO models. IWT models were also slightly less sensitive to bias introduced by overfitting and less sensitive to falsely identifying the existence of variability in treatment effect across the risk spectrum than ICO models. Conclusions Appropriately developed internal models produce relatively unbiased estimates of treatment effect across the spectrum of risk. When estimating treatment effect, internally developed risk models using both treatment arms should, in general, be preferred to models developed on the control population.

Individualised prediction of alternate-day aspirin treatment effects on the combined risk of cancer, cardiovascular disease and gastrointestinal bleeding in healthy women

“…[15][16][17][18][19][20] This would enable clinicians to estimate the response of an individual to aspirin prophylaxis and only treat those who are expected to benefit.…”

Development and Validation of a Model to Predict Absolute Vascular Risk Reduction by Moderate-Intensity Statin Therapy in Individual Patients With Type 2 Diabetes Mellitus

“…46 Moreover, it was recently shown that ARRs estimated with models that are internally developed in trial data result in limited bias compared with predictions based on existing external risk scores, given that the number of events per predictor does not exceed 10. 6 Also the fact that the newly developed model performs well in 2 external validation sets in which the patients differ considerably in baseline characteristics (Table 1) as well as in an observational cohort of patients with type 2 diabetes mellitus ( Figure I in the Data Supplement) is reassuring for the generalizability of this model to a broad range of patients with type 2 diabetes mellitus.…”

“…1 However disadvantages of subgroup analysis should be acknowledged, including that only 1 characteristic is studied at a time, whereas treatment effects are likely to be determined by a combination of patient characteristics. [5][6][7] Estimated treatment effects based on several patient characteristics, as well as the expected remaining risk when a patient is treated with a statin, might be more informative than average effects and recommendations from guidelines for both patients and clinicians. Therefore, we aimed to develop and validate a multivariable prediction model for ARR of major cardiovascular events by statin therapy for individual patients with type 2 diabetes mellitus.…”

Book Reviews

“…As such, its importance to a given reader depends on where you currently sit: if you think things are fine, I highly recommend this book (or, at least, Chapters 1, 2, 10, and 11 and the Preface). If you already agree that there is a problem, you might be better looking at the literature on ways around this problem, for example, Bretz and Westfall (2014), Burke et al (2014), Matsouaka, Li, and Cai (2014), Vickers, Kattan, and Sargent (2007), or many others. Due to the rapid growth of the complexity of computer systems in the past decades, reliable and high-quality computer software has become crucial in many fields including business, telecommunication, military, industrial process.…”