Authors' response to comments

Abstract

We are grateful to the authors who provided their insightful commentaries [1][2][3][4][5], which we hope will lead to more appropriate uses of the NRI and IDI metrics and their parent measures, the maximum relative utility, and discrimination slope. Here, we highlight common themes, clarify certain issue, and point out where we differ with some of the authors.Together with the papers they reference [6][7][8], several of the authors re-iterate the importance of model calibration when using the NRI and IDI metrics. Emphasis is appropriately placed on using smooth calibration plots to assess calibration. As suggested by van Calster et al.[9] and further illustrated by the theoretical examples in [6,7], 'moderate' (or 'level 3') calibration should be required: For persons with the same predicted risk, the observed event rate equals the predicted risk. As pointed out in [1], there is room for further discussion on recalibration and how and when it should be performed. When evaluating added value of new markers, it is reasonable to assess it on a model that is calibrated in the moderate sense: if a decision is made to use the new markers, they would be incorporated into a model for which satisfactory calibration needs to be ascertained before it is recommended for general applications. At the same time, we recognize the decision-making conundrum where in some applications to new data, we may not know ahead of time the extent of potential miscalibration.A key conclusion from the work of Chipman and Braun [6] is that calibration must be sufficiently met for the population of inference. If an investigator wishes to report the IDI in a clinically relevant subgroup, sufficient calibration must be established for that subgroup. If this is not the case, Simpson's Paradox may be observed in comparing the subgroup specific IDI to the IDI for the whole sample. Even small departures from calibration may lead to Simpson's Paradox; this further emphasizes the sensitivity of the discrimination slope to model calibration [2]. Of note, these issues are not resolved by using the weighted discrimination slope or IDI. While the weighted IDI removed the issue of Simpson's Paradox in simulations, under some settings, it yielded conclusions that were inconsistent with other measures including change in AUC, Brier Score, and other R-squared metrics [6]. Therefore, while illustrative, we do not recommend using it as a new metric.An important feature of the R-squared metrics is that, unlike the AUC, they integrate both calibration and discrimination into one number. Because the AUC is equivalent to the rank-based Mann-Whitney statistic, it necessarily loses information that relates to the calibration of the risk estimates. Thus, the AUC is a more pure measure of discrimination. The discrimination slope, on the other hand, retains all the information. Yet because the slope is not a proper scoring rule, it is susceptible to misleading conclusions for poorly calibrated models [6,7].

A Direct Comparison of the Relationship of Epigenetic Aging and Epigenetic Substance Consumption Markers to Mortality in the Framingham Heart Study

“…In this communication, we present four different metrics, Harrell’s C, pseudo R 2 , IDI and NRI, through which to analyze the effect of cg05575921, cg04987734, and LEA on survival prediction. To be clear, each of those metrics have strengths and weaknesses that are well noted in the literature [28,32,33,34]. Still, in each of those analyses, although the amount of additional information was rather small, LEA did seem to predict a small proportion of survival independent of the effects of cg05575921 and cg04987734.…”

Comments on “Net reclassification index at event rate: Properties and relationships”

“…So why not just use net benefit? Pencina et al state that in order to use net benefit, there need to be “well‐established thresholds,” and in their “vast and varied experience,” this is “rare.” If the authors are implying that there is often no single threshold, this is true, but irrelevant because net benefit is traditionally estimated across a range of thresholds. If the authors are implying that it is the suitable range of thresholds that is unknown, this is obviously false.…”

Single‐number summary and decision analytic measures can happily coexist

“…We argue that presenting standardized net benefit curves with maximum standardized net benefit as a companion single‐number summary offers a more elegant and interpretable pairing than standardized net benefit and the AUC. Moreover, Baker shows that the inverse of the NRI at event rate times the event rate can be interpreted as the summary test trade‐off, ie, an approximate lower bound over all thresholds for the minimum number of tests for a new marker that needs to be traded for a true positive to yield a positive net benefit …”