Discrimination and Calibration of Clinical Prediction Models

Alba, Ana C.; Agoritsas, Thomas; Walsh, Michael; Hanna, Steven; Iorio, Alfonso; Devereaux, P. J.; McGinn, Thomas; Guyatt, Gordon

doi:10.1001/jama.2017.12126

Cited by 1,037 publications

(1,010 citation statements)

References 39 publications

Supporting

Mentioning

940

Contrasting

Unclassified

Order By: Relevance

“…The area under the receiver operating characteristic curve (AUROC) is a useful indicator of a model’s ability to discriminate between those at high risk and those at lower risk of a given outcome [48]. The following are generally accepted interpretations of AUROC values: poor (< 0.60), possibly helpful (≥0.60 but < 0.70), acceptable (≥0.70 but < 0.80), excellent (≥0.80 but < 0.90), and outstanding (≥0.90) [48, 49].…”

Section: Area Under the Receiver Operating Characteristic Curvementioning

confidence: 99%

“…The following are generally accepted interpretations of AUROC values: poor (< 0.60), possibly helpful (≥0.60 but < 0.70), acceptable (≥0.70 but < 0.80), excellent (≥0.80 but < 0.90), and outstanding (≥0.90) [48, 49]. …”

Section: Area Under the Receiver Operating Characteristic Curvementioning

confidence: 99%

“…Specificity refers to a model’s ability to correctly rule out the individuals who do not have or will not develop the condition of interest [1]. The ideal model would produce 100% sensitivity and 100% specificity, but this is considered to be unrealistic [48]. …”

Section: Sensitivity Specificity and Risk Thresholdsmentioning

confidence: 99%

See 2 more Smart Citations

Prediction Models for Early Childhood Obesity: Applicability and Existing Issues

et al. 2018

View full text Add to dashboard Cite

Statistical models have been developed for the prediction or diagnosis of a wide range of outcomes. However, to our knowledge, only 7 published studies have reported models to specifically predict overweight and/or obesity in early childhood. These models were developed using known risk factors and vary greatly in terms of their discrimination and predictive capacities. There are currently no established guidelines on what constitutes an acceptable level of risk (i.e., risk threshold) for childhood obesity prediction models, but these should be set following consideration of the consequences of false-positive and false-negative predictions, as well as any relevant clinical guidelines. To date, no studies have examined the impact of using early childhood obesity prediction models as intervention tools. While these are potentially valuable to inform targeted interventions, the heterogeneity of the existing models and the lack of consensus on adequate thresholds limit their usefulness in practice.

show abstract

Section: Area Under the Receiver Operating Characteristic Curvementioning

confidence: 99%

Section: Area Under the Receiver Operating Characteristic Curvementioning

confidence: 99%

See 1 more Smart Citation

Prediction Models for Early Childhood Obesity: Applicability and Existing Issues

et al. 2018

View full text Add to dashboard Cite

show abstract

“…The performance of risk prediction models is divided into two components: discrimination and calibration. 6 Discrimination is the ability of the prediction model to distinguish whether high-risk patients are in fact low risk. 6 A well-known measurement tool of discrimination is the receiver-operating characteristic (ROC) curve or the C-statistic.…”

Section: Performance Of Prediction Modelsmentioning

confidence: 99%

“…6 When the calibration of a prediction model is poor, the model over-or underestimates the absolute probability of the outcome event, no matter how good the discrimination of a model is. Therefore, it is important to keep the calibration value as high as possible.…”

Section: Editorial Using Prediction Modelsmentioning

confidence: 99%

Prediction Models　― Why Are They Used or Not Used? ―

Nakamura

Nakai

2017

Circ J

View full text Add to dashboard Cite

models are complex. Performance of Prediction ModelsWe need to assess the performance of risk prediction models more carefully before applying their results in the clinical setting. The performance of risk prediction models is divided into two components: discrimination and calibration. 6 Discrimination is the ability of the prediction model to distinguish whether high-risk patients are in fact low risk. 6 A well-known measurement tool of discrimination is the receiver-operating characteristic (ROC) curve or the C-statistic. The C-statistic defines the probability that a randomly selected patient who developed an event had a higher risk score than a patient who had not developed the event. The ROC curve draws the plot of the sensitivity and (1−specificity) for all possible cutoff points (Figure 1). When the outcome is a binary event, the area under the ROC curve (AUC) is equivalent to the C-statistic, which ranges from 0.5 to 1, where 0.5 indicates no discriminatory ability and P redicting a patient's prognosis advances medical decision making in clinical settings. Risk prediction models (also called prognostic models, prediction rules, or risk scores) are tools to estimate individual patients' risk or probability by numerical values. Although many prediction models have been published, few have been used in routine clinical settings because of inconvenience and complexity. 1 In this issue of the Journal, Hu et al show how they applied the CHA2DS2-VASc score to predict the incidence of atrial fibrillation (AF) in chronic obstructive pulmonary disease (COPD) patients. 2 Previous studies have reported prediction models for AF based on community cohorts, 3-5 but most cardiologists are not familiar with these community-based prediction models. It seems reasonable to evaluate individual risk with what we already know, but we must be aware that CHA2DS2-VASc score was developed as the model for predicting ischemic stroke in patients with AF, not for incident AF in COPD patients. It is possible that applying the wrong prediction model may cause overor underestimation of a patient's risk. In order to reduce this, we need to understand the evaluation methods, risk prediction models and the reasons why high-performance Article p 1792The opinions expressed in this article are not necessarily those of the editors or of the Japanese Circulation Society.

show abstract

Using Free-Response Receiver Operating Characteristic Curves to Assess the Accuracy of Machine Diagnosis of Cancer

Moskowitz

2017

JAMA

View full text Add to dashboard Cite

In this issue of JAMA, Ehteshami Bejnordi et al 1 evaluate and compare the ability of 32 computer algorithms to identify the presence and location of metastatic lesions on pathology slide images of sentinel axillary lymph nodes from women with breast cancer. The authors used free-response receiver operating characteristic (FROC) curve analysis to assess diagnostic and localization accuracy. They found that the best algorithm performed similarly to a pathologist working without a time constraint (Figure).FROC analysis assesses the ability of a medical test to identify abnormalities on an image. Examples include identifying tumors in radiographs or foci of malignancy on histological slides. There are similarities between FROC analysis and the more commonly used receiver operating characteristic (ROC) curve analysis. 2,3 Conventional ROC curves, however, evaluate the accuracy of a test for detecting the presence or absence of disease but do not evaluate whether a test correctly identifies the location.

show abstract

Discrimination and Calibration of Clinical Prediction Models

Cited by 1,037 publications

References 39 publications

Prediction Models for Early Childhood Obesity: Applicability and Existing Issues

Prediction Models for Early Childhood Obesity: Applicability and Existing Issues

Prediction Models　― Why Are They Used or Not Used? ―

Using Free-Response Receiver Operating Characteristic Curves to Assess the Accuracy of Machine Diagnosis of Cancer

Contact Info

Product

Resources

About

Discrimination and Calibration of Clinical Prediction Models

Cited by 1,037 publications

References 39 publications

Prediction Models for Early Childhood Obesity: Applicability and Existing Issues

Prediction Models for Early Childhood Obesity: Applicability and Existing Issues

Prediction Models ― Why Are They Used or Not Used? ―

Using Free-Response Receiver Operating Characteristic Curves to Assess the Accuracy of Machine Diagnosis of Cancer

Contact Info

Product

Resources

About

Prediction Models　― Why Are They Used or Not Used? ―