Traditional Statistical Methods for Evaluating Prediction Models Are Uninformative as to Clinical Value: Towards a Decision Analytic Framework

Vickers, Andrew J.; Cronin, Angel M.

doi:10.1053/j.seminoncol.2009.12.004

Cited by 98 publications

(116 citation statements)

References 33 publications

Supporting

Mentioning

110

Contrasting

Unclassified

Order By: Relevance

“…Discrimination should not be reported in isolation because a poorly calibrated model can have the same discriminative capacity as a perfectly calibrated model. 29 One limitation of calibration is that assessment techniques do not allow for comparisons between models. In the validation cohorts, both the SHFM and the HFSS showed inadequate calibration attributable to the model overestimating survival in some groups of patients, including low-risk patients, blacks, and patients with ICD/CRT therapy.…”

Section: Discussionmentioning

confidence: 99%

“…Vickers et al 29 have proposed the use of simple decision analytic techniques to compare prediction models in terms of their consequences. These techniques weight true and false-positive errors differently, to reflect the impact of decision consequences (ie, risks associated with heart transplantation or ventricular assist device versus risks associated with continuing medical therapy).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Risk Prediction Models for Mortality in Ambulatory Patients With Heart Failure

Alba

Agoritsas

Jankowski

et al. 2013

Circ: Heart Failure

162

106

View full text Add to dashboard Cite

Background— Optimal management of heart failure requires accurate assessment of prognosis. Many prognostic models are available. Our objective was to identify studies that evaluate the use of risk prediction models for mortality in ambulatory patients with heart failure and describe their performance and clinical applicability. Methods and Results— We searched for studies in Medline, Embase, and CINAHL in May 2012. Two reviewers selected citations including patients with heart failure and reporting on model performance in derivation or validation cohorts. We abstracted data related to population, outcomes, study quality, model discrimination, and calibration. Of the 9952 studies reviewed, we included 34 studies testing 20 models. Only 5 models were validated in independent cohorts: the Heart Failure Survival Score, the Seattle Heart Failure Model, the PACE (incorporating peripheral vascular disease, age, creatinine, and ejection fraction) risk score, a model by Frankenstein et al, and the SHOCKED predictors. The Heart Failure Survival Score was validated in 8 cohorts (2240 patients), showing poor-to-modest discrimination ( c -statistic, 0.56–0.79), being lower in more recent cohorts. The Seattle Heart Failure Model was validated in 14 cohorts (16 057 patients), describing poor-to-acceptable discrimination (0.63–0.81), remaining relatively stable over time. Both models reported adequate calibration, although overestimating survival in specific populations. The other 3 models were validated in a cohort each, reporting poor-to-modest discrimination (0.66–0.74). Among the remaining 15 models, 6 were validated by bootstrapping ( c -statistic, 0.74–0.85); the rest were not validated. Conclusions— Externally validated heart failure models showed inconsistent performance. The Heart Failure Survival Score and Seattle Heart Failure Model demonstrated modest discrimination and questionable calibration. A new model derived from contemporary patient cohorts may be required for improved prognostic performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Risk Prediction Models for Mortality in Ambulatory Patients With Heart Failure

Alba

Agoritsas

Jankowski

et al. 2013

Circ: Heart Failure

162

106

View full text Add to dashboard Cite

show abstract

“…The goodness of fit or E/O ratio is commonly applied to measure how close the predicted and the observed values are [31,32]. The C-statistic is usually applied to measure how well the model will assign a higher probability of having an event to a case group and a lower probability to a non-case group [33].…”

Section: Discussionmentioning

confidence: 99%

Risk prediction models of breast cancer: a systematic review of model performances

Anothaisintawee

Teerawattananon

Wiratkapun

et al. 2011

Breast Cancer Res Treat

105

101

View full text Add to dashboard Cite

The number of risk prediction models has been increasingly developed, for estimating about breast cancer in individual women. However, those model performances are questionable. We therefore have conducted a study with the aim to systematically review previous risk prediction models. The results from this review help to identify the most reliable model and indicate the strengths and weaknesses of each model for guiding future model development. We searched MEDLINE (PubMed) from 1949 and EMBASE (Ovid) from 1974 until October 2010. Observational studies which constructed models using regression methods were selected. Information about model development and performance were extracted. Twenty-five out of 453 studies were eligible. Of these, 18 developed prediction models and 7 validated existing prediction models. Up to 13 variables were included in the models and sample sizes for each study ranged from 550 to 2,404,636. Internal validation was performed in four models, while five models had external validation. Gail and Rosner and Colditz models were the significant models which were subsequently modified by other scholars. Calibration performance of most models was fair to good (expected/observe ratio: 0.87-1.12), but discriminatory accuracy was poor to fair both in internal validation (concordance statistics: 0.53-0.66) and in external validation (concordance statistics: 0.56-0.63). Most models yielded relatively poor discrimination in both internal and external validation. This poor discriminatory accuracy of existing models might be because of a lack of knowledge about risk factors, heterogeneous subtypes of breast cancer, and different distributions of risk factors across populations. In addition the concordance statistic itself is insensitive to measure the improvement of discrimination. Therefore, the new method such as net reclassification index should be considered to evaluate the improvement of the performance of a new develop model.

show abstract

“…20,21 We performed 10-fold cross-validation techniques to estimate how the model will generalize to an independent population and correct for this optimism bias. 22 We assessed calibration performance by calculating the Hosmer-Lemeshow goodness-offit statistic, which measures whether the predicted probability of infection corresponds with the observed probability. A wellcalibrated model gives a corresponding P value greater than 0.05.…”

Section: Performance Measuresmentioning

confidence: 99%

Deriving and Validating A Risk Estimation Tool for Screening Asymptomatic Chlamydia and Gonorrhea

Falasinnu

Gilbert²,

Gustafson³

et al. 2014

Sexually Transmitted Diseases

View full text Add to dashboard Cite

Background: There has been considerable interest in the development of innovative service delivery modules for prioritizing resources in sexual health delivery in response to dwindling fiscal resources and rising infection rates.Methods: This study aims to derive and validate a risk scoring algorithm to accurately identify asymptomatic patients at increased risk for chlamydia and/or gonorrhea infection. We examined the electronic records of patient visits at sexual health clinics in Vancouver, Canada. We derived risk scores from regression coefficients of multivariable logistic regression model using visits between 2000 and 2006. We evaluated the model's discrimination, calibration, and screening performance. Temporal validation was assessed in visits from 2007 to 2012.Results: The prevalence of infection was 1.8% (n = 10,437) and 2.1% (n = 14,956) in the derivation and validation data sets, respectively. The final model included younger age, nonwhite ethnicity, multiple sexual partners, and previous infection and showed reasonable performance in the derivation (area under the receiver operating characteristic curve = 0.74; Hosmer-Lemeshow P = 0.91) and validation (area under the receiver operating characteristic curve = 0.64; Hosmer-Lemeshow P = 0.36) data sets. A risk score cutoff point of at least 6 detected 91% and 83% of cases by screening 68% and 68% of the derivation and validation populations, respectively.Conclusions: These findings support the use of the algorithm for individualized risk assessment and have important implications for reducing unnecessary screening and saving costs. Specifically, we anticipate that the algorithm has potential uses in alternative settings such as Internet-based testing contexts by facilitating personalized test recommendations, stimulating health care-seeking behavior, and aiding risk communication by increasing sexually transmitted infection risk perception through the creation of tailored risk messages to different groups.

show abstract

Traditional Statistical Methods for Evaluating Prediction Models Are Uninformative as to Clinical Value: Towards a Decision Analytic Framework

Cited by 98 publications

References 33 publications

Risk Prediction Models for Mortality in Ambulatory Patients With Heart Failure

Risk Prediction Models for Mortality in Ambulatory Patients With Heart Failure

Risk prediction models of breast cancer: a systematic review of model performances

Deriving and Validating A Risk Estimation Tool for Screening Asymptomatic Chlamydia and Gonorrhea

Contact Info

Product

Resources

About