Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: An observational cohort study

Abstract

Background The number of proposed prognostic models for COVID-19, which aim to predict disease outcomes, is growing rapidly. It is not known whether any are suitable for widespread clinical implementation. We addressed this question by independent and systematic evaluation of their performance among hospitalised COVID-19 cases. Methods We conducted an observational cohort study to assess candidate prognostic models, identified through a living systematic review. We included consecutive adults admitted to a secondary care hospital with PCR-confirmed or clinically diagnosed community-acquired COVID-19 (1st February to 30th April 2020). We reconstructed candidate models as per their original descriptions and evaluated performance for their original intended outcomes (clinical deterioration or mortality) and time horizons. We assessed discrimination using the area under the receiver operating characteristic curve (AUROC), and calibration using calibration plots, slopes and calibration-in-the-large. We calculated net benefit compared to the default strategies of treating all and no patients, and against the most discriminating predictor in univariable analyses, based on a limited subset of a priori candidates. Results We tested 22 candidate prognostic models among a cohort of 411 participants, of whom 180 (43.8%) and 115 (28.0%) met the endpoints of clinical deterioration and mortality, respectively. The highest AUROCs were achieved by the NEWS2 score for prediction of deterioration over 24 hours (0.78; 95% CI 0.73-0.83), and a novel model for prediction of deterioration <14 days from admission (0.78; 0.74-0.82). Calibration appeared generally poor for models that used probability outcomes. In univariable analyses, admission oxygen saturation on room air was the strongest predictor of in-hospital deterioration (AUROC 0.76; 0.71-0.81), while age was the strongest predictor of in-hospital mortality (AUROC 0.76; 0.71-0.81). No prognostic model demonstrated consistently higher net benefit than using the most discriminating univariable predictors to stratify treatment, across a range of threshold probabilities. Conclusions Oxygen saturation on room air and patient age are strong predictors of deterioration and mortality among hospitalised adults with COVID-19, respectively. None of the prognostic models evaluated offer incremental value for patient stratification to these univariable predictors.

Performance of prediction models for COVID-19: the Caudine Forks of the external validation

“…Moreover, the Authors found that (within their data) baseline oxygen saturation on room air was the most predictive variable for in-hospital worsening, while age was the most predictive variable for in-hospital mortality. Astonishingly, none of the 22 CPMs included in the external validation demonstrated significantly higher clinical utility when compared with these variables alone [4]. Together, these results imply that none of the selected CPMs should be recommended for adoption in daily clinical routine.…”

“…Some caution is required in the interpretation of the findings of Gupta et al [4]. First, this is a single site validation.…”

“…To address this gap in the current literature, a paper by Gupta et al [4], recently published in the European Respiratory Journal, aimed to externally validate 22 of the CPMs identified in the abovementioned systematic review [3]. Using data from 411 adults who were admitted to the University College Hospital in London with clinically diagnosed Covid-19, it showed that all CPMs performed poorly in the new data [4]. Moreover, the Authors found that (within their data) baseline oxygen saturation on room air was the most predictive variable for in-hospital worsening, while age was the most predictive variable for in-hospital mortality.…”

“…Indeed, despite the widespread interest in developing new CPMs, it is key that the research community takes stock of the available evidence, before adding more models to the mix. For example, the important study by Gupta et al [4] focussed on validating CPMs aimed to predict in-hospital clinical worsening or mortality among Covid-19 patients. Given that there are numerous other outcomes and different clinical settings (e.g., outpatient settings) where Covid-19 CPMs have been developed, future external validation studies should be a target for research in the near-future.…”

Improving the quality of machine learning in health applications and clinical research

“…Note that the implementation of any new algorithm is likely to influence the data routinely collected, by assigning increased importance in those features that are used as model inputs, while generating new information (that is, the model outcomes), thus completing the development cycle, and initiating a new cycle. comment subsequent validation studies have found simple bedside measurements outperform many of these models 11 . It is likely that at least some research waste could be avoided by full and transparent reporting, which helps to avoid pursuing predictive models that are unlikely to reach clinical use.…”