Event rate net reclassification index and the integrated discrimination improvement for studying incremental value of risk markers

Smeden, Maarten van; Moons, Karel G.M.

doi:10.1002/sim.7286

Cited by 10 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is a proper measure of global discrimination measure that serves as a measure of distance between the distributions of risk between events and non-events. A problem is that it may not be clinically relevant [26, 27]. If a model is calibrated in the large, so that the average predicted risk equals the observed event rate, the cutoff will be at the mean predicted risk.…”

Section: Risk Reclassificationmentioning

confidence: 99%

Quantifying the added value of new biomarkers: how and how not

2018

View full text Add to dashboard Cite

Over the past few decades, interest in biomarkers to enhance predictive modeling has soared. Methodology for evaluating these has also been an active area of research. There are now several performance measures available for quantifying the added value of biomarkers. This commentary provides an overview of methods currently used to evaluate new biomarkers, describes their strengths and limitations, and offers some suggestions on their use.Keywords: Biomarkers, Model fit, Calibration, Reclassification, Clinical utilityDuring the past few decades, there has been an explosion of work on the use of biomarkers in predictive modeling and whether it is useful to include these when evaluating risk of clinical events. As new biologic mechanisms have been discovered, genetic markers evolved, and new assays developed, questions about the usefulness of new markers for clinical prediction have been debated. In cardiology, several strong risk factors for cardiovascular disease, namely cholesterol levels, blood pressure, smoking, and diabetes, have been well-known for decades [1] and have been incorporated into clinical practice. They have also been included in predictive models for cardiovascular disease, primarily developed in the Framingham Heart Study [2]. Since then, many new markers with more modest effects have been discovered as new biologic pathways have been unearthed. In fields which have less powerful predictors to date, development and addition of predictive markers may be even more important.As interest in biomarkers has soared, so has the methodology used to evaluate their utility. There are now several performance measures available for quantifying the added value of biomarkers (Table 1), several of which have been proposed in the last decade. This commentary provides an overview of methods currently used to evaluate new biomarkers, describes their strengths and limitations, and offers some suggestions on their use. Likelihood functionsA fundamental construct for much of statistical modeling is the likelihood function. This reflects the probability, or "likelihood," of obtaining the observed data under the assumed model, including the selected variables and their associated parameters [3]. As more variables are added and the model fits the data better, the probability of obtaining the data that are actually observed improves. Much of statistical theory is based on this function. Thus, the primordial criterion of whether new variables, including biomarkers, can add to or improve a model is whether and by how much the likelihood increases. When the models are nested, we can test improvement with a likelihood ratio test, though other related tests, such as a Wald test, are sometimes used. For nonparametric models or machine learning tools, other loss functions are often used, such as cross-entropy or deviance, which are functions of the log likelihood for binary outcomes [4].Other likelihood-based measures do not directly perform a test of significance, but apply a penalty for added variables, such as the ...

show abstract

Section: Risk Reclassificationmentioning

confidence: 99%

Quantifying the added value of new biomarkers: how and how not

2018

View full text Add to dashboard Cite

show abstract

“…The above distinction explains why Kerr and Janes [3] object to a universal use of the event rate as a reasonable classification threshold, a concern also raised by others [2]. They remind us of the important decision-analytic relationship between the classification threshold and the misclassification costs for events and non-events.…”

mentioning

confidence: 94%

“…

…”

mentioning

confidence: 99%

“…If this is not the case, Simpson's Paradox may be observed in comparing the subgroup specific IDI to the IDI for the whole sample. Even small departures from calibration may lead to Simpson's Paradox; this further emphasizes the sensitivity of the discrimination slope to model calibration [2]. Of note, these issues are not resolved by using the weighted discrimination slope or IDI.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Authors' response to comments

Pencina

Chipman

Steyerberg

et al. 2017

Statistics in Medicine

View full text Add to dashboard Cite

We are grateful to the authors who provided their insightful commentaries [1][2][3][4][5], which we hope will lead to more appropriate uses of the NRI and IDI metrics and their parent measures, the maximum relative utility, and discrimination slope. Here, we highlight common themes, clarify certain issue, and point out where we differ with some of the authors.Together with the papers they reference [6][7][8], several of the authors re-iterate the importance of model calibration when using the NRI and IDI metrics. Emphasis is appropriately placed on using smooth calibration plots to assess calibration. As suggested by van Calster et al.[9] and further illustrated by the theoretical examples in [6,7], 'moderate' (or 'level 3') calibration should be required: For persons with the same predicted risk, the observed event rate equals the predicted risk. As pointed out in [1], there is room for further discussion on recalibration and how and when it should be performed. When evaluating added value of new markers, it is reasonable to assess it on a model that is calibrated in the moderate sense: if a decision is made to use the new markers, they would be incorporated into a model for which satisfactory calibration needs to be ascertained before it is recommended for general applications. At the same time, we recognize the decision-making conundrum where in some applications to new data, we may not know ahead of time the extent of potential miscalibration.A key conclusion from the work of Chipman and Braun [6] is that calibration must be sufficiently met for the population of inference. If an investigator wishes to report the IDI in a clinically relevant subgroup, sufficient calibration must be established for that subgroup. If this is not the case, Simpson's Paradox may be observed in comparing the subgroup specific IDI to the IDI for the whole sample. Even small departures from calibration may lead to Simpson's Paradox; this further emphasizes the sensitivity of the discrimination slope to model calibration [2]. Of note, these issues are not resolved by using the weighted discrimination slope or IDI. While the weighted IDI removed the issue of Simpson's Paradox in simulations, under some settings, it yielded conclusions that were inconsistent with other measures including change in AUC, Brier Score, and other R-squared metrics [6]. Therefore, while illustrative, we do not recommend using it as a new metric.An important feature of the R-squared metrics is that, unlike the AUC, they integrate both calibration and discrimination into one number. Because the AUC is equivalent to the rank-based Mann-Whitney statistic, it necessarily loses information that relates to the calibration of the risk estimates. Thus, the AUC is a more pure measure of discrimination. The discrimination slope, on the other hand, retains all the information. Yet because the slope is not a proper scoring rule, it is susceptible to misleading conclusions for poorly calibrated models [6,7].

show abstract

“…However, NRI has its disadvantages: NRI only considers performance improvement at one time point and fail to evaluate the overall improvement of a predictive model. Therefore, we could use another index: IDI (Integrated Discrimination Index) (40,41). Some readers may ask: is AUC/C-statistics able to evaluate the overall improvement of a predictive model?…”

Section: Introductionmentioning

confidence: 99%

In-depth mining of clinical data: the construction of clinical prediction model with R

et al. 2019

View full text Add to dashboard Cite

This article is the series of methodology of clinical prediction model construction (total 16 sections of this methodology series). The first section mainly introduces the concept, current application status, construction methods and processes, classification of clinical prediction models, and the necessary conditions for conducting such researches and the problems currently faced. The second episode of these series mainly concentrates on the screening method in multivariate regression analysis. The third section mainly introduces the construction method of prediction models based on Logistic regression and Nomogram drawing. The fourth episode mainly concentrates on Cox proportional hazards regression model and Nomogram drawing. The fifth Section of the series mainly introduces the calculation method of C-Statistics in the logistic regression model. The sixth section mainly introduces two common calculation methods for C-Index in Cox regression based on R. The seventh section focuses on the principle and calculation methods of Net Reclassification Index (NRI) using R. The eighth section focuses on the principle and calculation methods of IDI (Integrated Discrimination Index) using R. The ninth section continues to explore the evaluation method of clinical utility after predictive model construction: Decision Curve Analysis. The tenth section is a supplement to the previous section and mainly introduces the Decision Curve Analysis of survival outcome data. The eleventh section mainly discusses the external validation method of Logistic regression model. The twelfth mainly discusses the in-depth evaluation of Cox regression model based on R, including calculating the concordance index of discrimination (C-index) in the validation data set and drawing the calibration curve. The thirteenth section mainly introduces how to deal with the survival data outcome using competitive risk model with R. The fourteenth section mainly introduces how to draw the nomogram of the competitive risk model with R. The fifteenth section of the series mainly discusses the identification of outliers and the interpolation of missing values. The sixteenth section of the series mainly introduced the Zhou et al. Clinical prediction models with R

show abstract

Event rate net reclassification index and the integrated discrimination improvement for studying incremental value of risk markers

Cited by 10 publications

References 20 publications

Quantifying the added value of new biomarkers: how and how not

Quantifying the added value of new biomarkers: how and how not

Authors' response to comments

In-depth mining of clinical data: the construction of clinical prediction model with R

Contact Info

Product

Resources

About