Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: an observational cohort study

Abstract

BackgroundThe number of proposed prognostic models for COVID-19 is growing rapidly, but it is unknown whether any are suitable for widespread clinical implementation.MethodsWe independently externally validated the performance candidate prognostic models, identified through a living systematic review, among consecutive adults admitted to hospital with a final diagnosis of COVID-19. We reconstructed candidate models as per original descriptions and evaluated performance for their original intended outcomes using predictors measured at admission. We assessed discrimination, calibration and net benefit, compared to the default strategies of treating all and no patients, and against the most discriminating predictor in univariable analyses.ResultsWe tested 22 candidate prognostic models among 411 participants with COVID-19, of whom 180 (43.8%) and 115 (28.0%) met the endpoints of clinical deterioration and mortality, respectively. Highest areas under receiver operating characteristic (AUROC) curves were achieved by the NEWS2 score for prediction of deterioration over 24 h (0.78; 95% CI 0.73–0.83), and a novel model for prediction of deterioration <14 days from admission (0.78; 0.74–0.82). The most discriminating univariable predictors were admission oxygen saturation on room air for in-hospital deterioration (AUROC 0.76; 0.71–0.81), and age for in-hospital mortality (AUROC 0.76; 0.71–0.81). No prognostic model demonstrated consistently higher net benefit than these univariable predictors, across a range of threshold probabilities.ConclusionsAdmission oxygen saturation on room air and patient age are strong predictors of deterioration and mortality among hospitalised adults with COVID-19, respectively. None of the prognostic models evaluated here offered incremental value for patient stratification to these univariable predictors.

Development and validation of the patient history COVID-19 (PH-Covid19) scoring system: a multivariable prediction model of death in Mexican patients with COVID-19

“…Similar to other studies, we found that older age is the main risk factor for dying from COVID-19, with every 10-year increase associated with the largest increases in the HR [11,14,23]. In a recent prospective validation study of prognostic models for COVID-19, none of the models that predicted mortality in patients with COVID-19 were better than age alone to predict in-hospital mortality [24]. This may be explained by the fact that many models for COVID-19 have failed to account for the large increases in mortality risk due to increasing ages.…”

Development and validation of the 4C Deterioration model for adults hospitalised with COVID-19

“…We only included participants admitted or first assessed for COVID-19 until 26/08/2020 in order to allow a minimum four-week interval for registration of outcome events (until final data extraction date of 24/09/2020). Participants who had ongoing hospital care at the end of follow-up were classified as not meeting the endpoint, since the risk of deterioration declines with time since admission 8 . Missing outcomes were imputed, as described below.…”

“…Decision curve analysis was also done in the internal-external cross-validation validation sets to quantify the net benefit of implementing the model in clinical practice 28 , compared to: (a) a 'treat all' approach; (b) a 'treat none' approach; and (c) using other candidate generic and COVID-specific clinical prognostic models to stratify treatment, identified by recent systematic reviews 6,8,9 . Only candidate models where constituent variables were available among >60% of the cohort were considered.…”

“…The vast majority of candidate models subjected to comprehensive quality assessment have been classified as being at high risk of bias, and therefore may not be generalisable 6,7 . Moreover, none of the multivariable prognostic models included in a systematic head-to-head external validation study outperformed univariable predictors 8 , highlighting a critical need to adhere to rigorous model development methodology using large scale multi-site data, in order to facilitate generalisability.…”

SARS-CoV-2 RNAemia and proteomic biomarker trajectory inform prognostication in COVID-19 patients admitted to intensive care

“…Coronavirus disease 2019 (COVID-19) caused by the severe acute respiratory syndrome coronavirus (SARS-CoV-2; a single-stranded RNA virus) poses an unprecedented challenge to health care systems globally. It is increasingly apparent that conventional prognostic scores for critically ill patients admitted to intensive care units (ICUs) such as the APACHE II (Acute Physiology and Chronic Health Evaluation) score 1 and SOFA (Sequential Organ Failure Assessment) score 2 , are not discriminatory in COVID-19 ICU patients [3][4][5][6] .…”