The area under the receiver operating characteristic (ROC) curve (AUC) is commonly used for assessing the discriminative ability of prediction models even though the measure is criticized for being clinically irrelevant and lacking an intuitive interpretation. Every tutorial explains how the coordinates of the ROC curve are obtained from the risk distributions of diseased and non-diseased individuals, but it has not become common sense that therewith the ROC plot is just another way of presenting these risk distributions. We show how the ROC curve is an alternative way to present risk distributions of diseased and non-diseased individuals and how the shape of the ROC curve informs about the overlap of the risk distributions. For example, ROC curves are rounded when the prediction model included variables with similar effect on disease risk and have an angle when, for example, one binary risk factor has a stronger effect; and ROC curves are stepped rather than smooth when the sample size or incidence is low, when the prediction model is based on a relatively small set of categorical predictors. This alternative perspective on the ROC plot invalidates most purported limitations of the AUC and attributes others to the underlying risk distributions. AUC is a measure of the discriminative ability of prediction models. The assessment of prediction models should be supplemented with other metrics to assess their clinical utility.
professional education on the utility and limitations of pharmacogenetic testing was desired by most stakeholders.Conclusion: While the evidence for DPD testing was sufficient, only after the update of a National guideline and local consensus meetings the proportion of FP users that were DPD tested pretreatment rose to 87%. The implementation of personalized medicine requires stakeholders involved to attune practice, culture and structure.
PurposeThe area under the receiver operating characteristic curve (AUC) is
commonly used for evaluating the improvement of polygenic risk models and
increasingly assessed together with the net reclassification improvement
(NRI) and integrated discrimination improvement (IDI). We evaluated how
researchers described and interpreted AUC, NRI, and IDI when simultaneously
assessed.MethodsWe reviewed how researchers described definitions of AUC, NRI and IDI
and how they computed each metric. Next, we reviewed how the increment in
AUC, NRI and IDI were interpreted; and how the overall conclusion about the
improvement of the risk model was reached.ResultsAUC, NRI and IDI were correctly defined in 63%, 70%, and 0% of the
articles. All statistically significant values and almost half of the
non-significant were interpreted as indicative of improvement, irrespective
of the values of the metrics. Also, small, nonsignificant changes in the AUC
were interpreted as indication of improvement when NRI and IDI were
statistically significant.ConclusionResearchers have insufficient knowledge about how to interpret the
various metrics for the assessment of the predictive performance of
polygenic risk models and rely on the statistical significance for their
interpretation. A better understanding is needed to achieve more meaningful
interpretation of polygenic prediction studies.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.