Simpson's paradox in the integrated discrimination improvement

Abstract

The Integrated Discrimination Improvement (IDI) is commonly used to compare two risk prediction models; it summarizes the extent a new model increases risk in events and decreases risk in non-events. The IDI averages risks across events and non-events and is therefore susceptible to Simpson's Paradox. In some settings, adding a predictive covariate to a well calibrated model results in an overall negative (positive) IDI. However, if stratified by that same covariate, the strata-specific IDIs are positive (negative). Meanwhile, the calibration (observed to expected ratio, O/E, and Hosmer-Lemeshow Goodness of Fit Test, GOF), area under the receiver-operating-characteristic curve (AUC), and Brier score improve overall and by stratum. We ran extensive simulations to investigate the impact of an imbalanced covariate upon metrics (IDI, AUC, Brier score, and R2), provide an analytic explanation for the paradox in the IDI, and use an investigative metric, a Weighted IDI, to better understand the paradox. In simulations, all instances of the paradox occurred under stratum-specific mis-calibration, yet there were mis-calibrated settings in which the paradox did not occur. The paradox is illustrated on Cancer Genomics Network data by calculating predictions based on two versions of BRCAPRO, a Mendelian risk prediction model for breast and ovarian cancer. In both simulations and the CGN data, overall model calibration did not guarantee stratum-level calibration. We conclude that the IDI should only assess model performance among a clinically relevant subset when stratum-level calibration is strictly met and recommend calculating additional metrics to confirm the direction and conclusions of the IDI.

Authors' response to comments

“…Of note, these issues are not resolved by using the weighted discrimination slope or IDI. While the weighted IDI removed the issue of Simpson's Paradox in simulations, under some settings, it yielded conclusions that were inconsistent with other measures including change in AUC, Brier Score, and other R-squared metrics [6]. Therefore, while illustrative, we do not recommend using it as a new metric.An important feature of the R-squared metrics is that, unlike the AUC, they integrate both calibration and discrimination into one number.…”

“…

We are grateful to the authors who provided their insightful commentaries [1][2][3][4][5], which we hope will lead to more appropriate uses of the NRI and IDI metrics and their parent measures, the maximum relative utility, and discrimination slope. Here, we highlight common themes, clarify certain issue, and point out where we differ with some of the authors.Together with the papers they reference [6][7][8], several of the authors re-iterate the importance of model calibration when using the NRI and IDI metrics. Emphasis is appropriately placed on using smooth calibration plots to assess calibration.

…”

“…Emphasis is appropriately placed on using smooth calibration plots to assess calibration. As suggested by van Calster et al[9] and further illustrated by the theoretical examples in [6,7], 'moderate' (or 'level 3') calibration should be required: For persons with the same predicted risk, the observed event rate equals the predicted risk. As pointed out in [1], there is room for further discussion on recalibration and how and when it should be performed.…”

“…When evaluating added value of new markers, it is reasonable to assess it on a model that is calibrated in the moderate sense: if a decision is made to use the new markers, they would be incorporated into a model for which satisfactory calibration needs to be ascertained before it is recommended for general applications. At the same time, we recognize the decision-making conundrum where in some applications to new data, we may not know ahead of time the extent of potential miscalibration.A key conclusion from the work of Chipman and Braun [6] is that calibration must be sufficiently met for the population of inference. If an investigator wishes to report the IDI in a clinically relevant subgroup, sufficient calibration must be established for that subgroup.…”

“…The discrimination slope, on the other hand, retains all the information. Yet because the slope is not a proper scoring rule, it is susceptible to misleading conclusions for poorly calibrated models [6,7].…”