Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers

Abstract

Summary Appropriate quantification of added usefulness offered by new markers included in risk prediction algorithms is a problem of active research and debate. Standard methods, including statistical significance and c statistic are useful but not sufficient. Net reclassification improvement (NRI) offers a simple intuitive way of quantifying improvement offered by new markers and has been gaining popularity among researchers. However, several aspects of the NRI have not been studied in sufficient detail. In this paper we propose a prospective formulation for the NRI which offers immediate application to survival and competing risk data as well as allows for easy weighting with observed or perceived costs. We address the issue of the number and choice of categories and their impact on NRI. We contrast category-based NRI with one which is category-free and conclude that NRIs cannot be compared across studies unless they are defined in the same manner. We discuss the impact of differing event rates when models are applied to different samples or definitions of events and durations of follow-up vary between studies. We also show how NRI can be applied to case-control data. The concepts presented in the paper are illustrated in a Framingham Heart Study example. In conclusion, NRI can be readily calculated for survival, competing risk, and case-control data, is more objective and comparable across studies using the category-free version, and can include relative costs for classifications. We recommend that researchers clearly define and justify the choices they make when choosing NRI for their application.

Protein Biomarkers of Cardiovascular Disease and Mortality in the Community

“…Previous studies, however, did not demonstrate any relation to CVD outcomes 10, 21. Adipsin has been shown to regulate inflammation in adipose tissue and may improve beta‐cell function in diabetes mellitus 11, 23, 24. Among women, adipsin is correlated with metabolic parameters, including body mass index, insulin resistance, and carotid atherosclerosis,3, 12, 25 but it has not been linked to adverse CVD outcomes previously.…”

“…We also calculated integrated discrimination improvement and net reclassification improvement metrics 11. For net reclassification improvement, we used 2 categories with the threshold set at observed event rate.…”

Soluble ST2 predicts outcome and hemorrhagic transformation after acute stroke

“…We evaluated the ability of sST2 to reclassify risk by comparing the prognostic accuracy of traditional clinical risk factors and calculating the net reclassification improvement (NRI) and integrated discrimination improvement (IDI) for models enriched with sST2. We calculated continuous (category‐free) NRI values in groups, since there were no prespecified, externally validated risk categories 18, 19. Kaplan–Meier survival curves were implemented to further investigate the ability of sST2 to predict mortality.…”

Performance of AHEAD Score in an Asian Cohort of Acute Heart Failure With Either Preserved or Reduced Left Ventricular Systolic Function

“…The Akaike information criterion was then used to compare the model fit of prognostic scoring systems. Improvement of prognostic accuracy from the new scoring index was assessed by calculating the net reclassification improvement with category‐free and category‐based methods,12 using the original AHEAD index as the reference. Another cohort of AHF13 was enrolled from January 2013 to September 2015 to validate the new prediction model.…”