Large-scale external validation and comparison of prognostic models: an application to chronic obstructive pulmonary disease

Abstract

BackgroundExternal validations and comparisons of prognostic models or scores are a prerequisite for their use in routine clinical care but are lacking in most medical fields including chronic obstructive pulmonary disease (COPD). Our aim was to externally validate and concurrently compare prognostic scores for 3-year all-cause mortality in mostly multimorbid patients with COPD.MethodsWe relied on 24 cohort studies of the COPD Cohorts Collaborative International Assessment consortium, corresponding to primary, secondary, and tertiary care in Europe, the Americas, and Japan. These studies include globally 15,762 patients with COPD (1871 deaths and 42,203 person years of follow-up). We used network meta-analysis adapted to multiple score comparison (MSC), following a frequentist two-stage approach; thus, we were able to compare all scores in a single analytical framework accounting for correlations among scores within cohorts. We assessed transitivity, heterogeneity, and inconsistency and provided a performance ranking of the prognostic scores.ResultsDepending on data availability, between two and nine prognostic scores could be calculated for each cohort. The BODE score (body mass index, airflow obstruction, dyspnea, and exercise capacity) had a median area under the curve (AUC) of 0.679 [1st quartile–3rd quartile = 0.655–0.733] across cohorts. The ADO score (age, dyspnea, and airflow obstruction) showed the best performance for predicting mortality (difference AUCADO – AUCBODE = 0.015 [95% confidence interval (CI) = −0.002 to 0.032]; p = 0.08) followed by the updated BODE (AUCBODE updated – AUCBODE = 0.008 [95% CI = −0.005 to +0.022]; p = 0.23). The assumption of transitivity was not violated. Heterogeneity across direct comparisons was small, and we did not identify any local or global inconsistency.ConclusionsOur analyses showed best discriminatory performance for the ADO and updated BODE scores in patients with COPD. A limitation to be addressed in future studies is the extension of MSC network meta-analysis to measures of calibration. MSC network meta-analysis can be applied to prognostic scores in any medical field to identify the best scores, possibly paving the way for stratified medicine, public health, and research.Electronic supplementary materialThe online version of this article (10.1186/s12916-018-1013-y) contains supplementary material, which is available to authorized users.

Prognostic models for outcome prediction in patients with chronic obstructive pulmonary disease: systematic review and critical appraisal

“…An essential step before the application of prediction models in clinical practice is their external validation in independent populations with different clinical characteristics and comparison of performance among different prediction models to identify the models with the best discrimination and calibration. A large scale effort to externally validate and compare multiple prognostic models for COPD patients was recently published 100. The researchers used network meta-analysis to compare the performance of eight multivariable prognostic models and two different GOLD classifications in 24 cohort studies.…”

“…In this analysis, the updated ADO index had the best ability to predict three year mortality in patients with COPD, followed by the updated BODE index and e-BODE index. However, the researchers pointed out that the approach of network meta-analysis has not yet integrated the synthesis of calibration measures 100…”

Prediction of five-year mortality after COPD diagnosis using primary care records

“…However, existing prognosis models focus on prevalent COPD, rather than incident cases, meaning that they depend on variables which are often not recorded in GP records at the time of COPD diagnosis. Additionally, external validation of these models appears to be rare, and when performed have resulted in inconsistent findings [ 3 – 5 ].…”

The stability of the ADO score among UK COPD patients from The Health Improvement Network

“…Multicomponent prognostic scores can better evaluate the risk of deterioration or death compared with FEV 1 alone as they combine multiple domains of COPD. The ADO (age, dyspnoea, airflow obstruction) score combines three easily accessible components and accurately predicts 3-year mortality [9,10].…”