Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study

Abstract

Objective To create and validate a simple and transferable machine learning model from electronic health record data to accurately predict clinical deterioration in patients with covid-19 across institutions, through use of a novel paradigm for model development and code sharing. Design Retrospective cohort study. Setting One US hospital during 2015-21 was used for model training and internal validation. External validation was conducted on patients admitted to hospital with covid-19 at 12 other US medical centers during 2020-21. Participants 33 119 adults (≥18 years) admitted to hospital with respiratory distress or covid-19. Main outcome measures An ensemble of linear models was trained on the development cohort to predict a composite outcome of clinical deterioration within the first five days of hospital admission, defined as in-hospital mortality or any of three treatments indicating severe illness: mechanical ventilation, heated high flow nasal cannula, or intravenous vasopressors. The model was based on nine clinical and personal characteristic variables selected from 2686 variables available in the electronic health record. Internal and external validation performance was measured using the area under the receiver operating characteristic curve (AUROC) and the expected calibration error—the difference between predicted risk and actual risk. Potential bed day savings were estimated by calculating how many bed days hospitals could save per patient if low risk patients identified by the model were discharged early. Results 9291 covid-19 related hospital admissions at 13 medical centers were used for model validation, of which 1510 (16.3%) were related to the primary outcome. When the model was applied to the internal validation cohort, it achieved an AUROC of 0.80 (95% confidence interval 0.77 to 0.84) and an expected calibration error of 0.01 (95% confidence interval 0.00 to 0.02). Performance was consistent when validated in the 12 external medical centers (AUROC range 0.77-0.84), across subgroups of sex, age, race, and ethnicity (AUROC range 0.78-0.84), and across quarters (AUROC range 0.73-0.83). Using the model to triage low risk patients could potentially save up to 7.8 bed days per patient resulting from early discharge. Conclusion A model to predict clinical deterioration was developed rapidly in response to the covid-19 pandemic at a single hospital, was applied externally without the sharing of data, and performed well across multiple medical centers, patient subgroups, and time periods, showing its potential as a tool for use in optimizing healthcare resources.

Augmenting existing deterioration indices with chest radiographs to predict clinical deterioration

“…We describe a new approach for augmenting EHR-based deterioration models with imaging data. We applied the approach to two EHR-based deterioration indices used for COVID-19, EDI and a locally-developed model called MCURES [ 4 , 5 ]. When applied to data from a large hospital, the image-augmented deterioration indices identified over five times more low-risk COVID-19 patients than the deterioration indices based on EHR data alone.…”

“…In the analysis, we augmented two different existing EHR-based deterioration indices: the Epic Deterioration Index (EDI) and MCURES [ 4 , 5 ].…”

“…While in-hospital deterioration models have been largely designed to identify patients at high risk for acute in-hospital complications [ 2 , 3 ], one potential novel application of these indices is identifying subsets of patients at low-risk of severe outcomes for safe, early discharge to alleviate hospital strain. For example, a recent validation study of the EPIC deterioration index found that it had promising predictive performance for identifying subsets of high- and low- risk patients with COVID-19 [ 4 , 5 ].…”

“…In this work, we describe a method using machine learning to augment existing deterioration indices with chest radiographs. We apply the approach to predict deterioration of patients hospitalized with COVID19, augmenting two existing EHR-based deterioration indices [ 4 , 5 ]. While focusing on the clinical problem of COVID-19, this approach has broader applicability to other risk models that could be augmented with imaging data.…”

Progress against health inequities is slow and obstacles are numerous

“…In a BMJ special paper, Tang and colleagues report on the development and validation of a model to predict clinical deterioration in patients admitted to hospital with covid-19 (doi:10.1136/bmj-2021-068576). 10 The novelty of the paper is not the algorithm but rather the methods used to create it. The machine learning model is open source.…”