Evaluating the Ability of Economic Models of Diabetes to Simulate New Cardiovascular Outcomes Trials: A Report on the Ninth Mount Hood Diabetes Challenge

Abstract

The cardiovascular outcomes challenge examined the predictive accuracy of 10 diabetes models in estimating hard outcomes in 2 recent cardiovascular outcomes trials (CVOTs) and whether recalibration can be used to improve replication. Methods: Participating groups were asked to reproduce the results of the Empagliflozin Cardiovascular Outcome Event Trial in Type 2 Diabetes Mellitus Patients (EMPA-REG OUTCOME) and the Canagliflozin Cardiovascular Assessment Study (CANVAS) Program. Calibration was performed and additional analyses assessed model ability to replicate absolute event rates, hazard ratios (HRs), and the generalizability of calibration across CVOTs within a drug class. Results: Ten groups submitted results. Models underestimated treatment effects (ie, HRs) using uncalibrated models for both trials. Calibration to the placebo arm of EMPA-REG OUTCOME greatly improved the prediction of event rates in the placebo, but less so in the active comparator arm. Calibrating to both arms of EMPA-REG OUTCOME individually enabled replication of the observed outcomes. Using EMPA-REG OUTCOME-calibrated models to predict CANVAS Program outcomes was an improvement over uncalibrated models but failed to capture treatment effects adequately. Applying canagliflozin HRs directly provided the best fit. Conclusions: The Ninth Mount Hood Diabetes Challenge demonstrated that commonly used risk equations were generally unable to capture recent CVOT treatment effects but that calibration of the risk equations can improve predictive accuracy. Although calibration serves as a practical approach to improve predictive accuracy for CVOT outcomes, it does not extrapolate generally to other settings, time horizons, and comparators. New methods and/or new risk equations for capturing these CV benefits are needed.

Cardiovascular risk prediction in type 2 diabetes: a comparison of 22 risk scores in primary care setting

“…With over 300+ published CVD risk prediction tools 5 , many of which have not been validated in T2DM patients, nor directly compared within the same patient population, it is unclear which CVD scores performs best in T2DM. Previous comparisons only partially addressed this question, due to either focusing on non-representative T2DM patient enrolled in drug trails 6 , focused on a relatively short follow-up 7 , or used a very modest sample of T2DM patients 8 , and often focusing on a small subset of available scores utilized in clinical practice, without exploring performance to predict CVD outcomes more relevant for T2DM patients. Quite apart from the greater CVD risk, even at a given level of individual risk factors, it is evident that the initial presentation of CVD in T2DM differs from that of the general population, with greater representation of heart failure and of peripheral artery disease (PAD), while hemorrhagic strokes are less frequent 9 .…”

Development and validation of the CHIME simulation model to assess lifetime health outcomes of prediabetes and type 2 diabetes in Chinese populations: A modeling study

“…We identified previous diabetes outcome models registered in Mount Hood Diabetes registry of simulation models [ 31 ]; participants characteristics, model development, and validation strategies are detailed in S1 Table . Almost all identified models were proprietary and did not have publicly available code; only the UKPDS-OM2 and RECODe had user interfaces for comparative performance assessment.…”

Prediction of mortality and major cardiovascular complications in type 2 diabetes: External validation of UK Prospective Diabetes Study outcomes model version 2 in two European observational cohorts

“…The model, as described by Hayes et al 2 is transparent 3,4 and has been validated internally as well as externally. 3,5 The UKPDS-OM2 has significant advantages over version 1, as it is based on longer follow-up data (almost double follow-up time), simulates more outcomes, and captures more comprehensively the progression of diabetes. 2,6 In view of a potential wide utilization of the UKPDS-OM2 in cost-effectiveness analysis and in the evaluation of strategies for the management of T2D at the European level in the future, external validation in data across European countries is of great interest.…”

“…The model predicts several types of complications common to people with T2D, using Monte Carlo simulation and risk equations fitted on the UK Prospective Diabetes Study (UKPDS) data that account for a range of patient characteristics, such as age, sex, preexisting complications, laboratory values and lifestyle habits. The model, as described by Hayes et al 2 is transparent 3,4 and has been validated internally as well as externally 3,5 . The UKPDS‐OM2 has significant advantages over version 1, as it is based on longer follow‐up data (almost double follow‐up time), simulates more outcomes, and captures more comprehensively the progression of diabetes 2,6 …”