Development and External Validation of a Prognostic Tool for COVID-19 Critical Disease

Chow, Daniel; Glavis-Bloom, Justin; Soun, Jennifer E.; Weinberg, Brent D.; Loveless, Theresa B; Xie, Xiaohui; Mutasa, Simukayi; Monuki, Edwin S.; Park, Jung In; Bota, Daniela; Wu, Jie; Thompson, Leslie M.; Boden‐Albala, Bernadette; Khan, Saahir; Amin, Alpesh; Chang, Peter D.

doi:10.1101/2020.05.06.20093435

Cited by 3 publications

(1 citation statement)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many predictive models have met with success; however, these models only consider demographics, clinical symptoms, or laboratory values rather than considering all these factors conjointly [10][11][12][13][14][15][16][17]. More recent studies have accounted for fundamental aspects of machine learning but are limited in scope [13,[18][19][20][21][22]. These studies lack either temporal benchmarks, interhospital or prospective validation, systematic evaluation of multiple models, consideration of covariate correlations, or assessment of the impact of the imputed data.…”

Section: Introductionmentioning

confidence: 99%

Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation (Preprint)

Vaid¹,

Somani²,

Russak³

et al. 2020

Preprint

View full text Add to dashboard Cite

BACKGROUND COVID-19 has infected millions of people worldwide and is responsible for several hundred thousand fatalities. The COVID-19 pandemic has necessitated thoughtful resource allocation and early identification of high-risk patients. However, effective methods to meet these needs are lacking. OBJECTIVE The aims of this study were to analyze the electronic health records (EHRs) of patients who tested positive for COVID-19 and were admitted to hospitals in the Mount Sinai Health System in New York City; to develop machine learning models for making predictions about the hospital course of the patients over clinically meaningful time horizons based on patient characteristics at admission; and to assess the performance of these models at multiple hospitals and time points. METHODS We used Extreme Gradient Boosting (XGBoost) and baseline comparator models to predict in-hospital mortality and critical events at time windows of 3, 5, 7, and 10 days from admission. Our study population included harmonized EHR data from five hospitals in New York City for 4098 COVID-19–positive patients admitted from March 15 to May 22, 2020. The models were first trained on patients from a single hospital (n=1514) before or on May 1, externally validated on patients from four other hospitals (n=2201) before or on May 1, and prospectively validated on all patients after May 1 (n=383). Finally, we established model interpretability to identify and rank variables that drive model predictions. RESULTS Upon cross-validation, the XGBoost classifier outperformed baseline models, with an area under the receiver operating characteristic curve (AUC-ROC) for mortality of 0.89 at 3 days, 0.85 at 5 and 7 days, and 0.84 at 10 days. XGBoost also performed well for critical event prediction, with an AUC-ROC of 0.80 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. In external validation, XGBoost achieved an AUC-ROC of 0.88 at 3 days, 0.86 at 5 days, 0.86 at 7 days, and 0.84 at 10 days for mortality prediction. Similarly, the unimputed XGBoost model achieved an AUC-ROC of 0.78 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. Trends in performance on prospective validation sets were similar. At 7 days, acute kidney injury on admission, elevated LDH, tachypnea, and hyperglycemia were the strongest drivers of critical event prediction, while higher age, anion gap, and C-reactive protein were the strongest drivers of mortality prediction. CONCLUSIONS We externally and prospectively trained and validated machine learning models for mortality and critical events for patients with COVID-19 at different time horizons. These models identified at-risk patients and uncovered underlying relationships that predicted outcomes.

show abstract

Section: Introductionmentioning

confidence: 99%

Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation (Preprint)

Vaid¹,

Somani²,

Russak³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation

Vaid¹,

Somani²,

Russak³

et al. 2020

J Med Internet Res

195

224

View full text Add to dashboard Cite

Background COVID-19 has infected millions of people worldwide and is responsible for several hundred thousand fatalities. The COVID-19 pandemic has necessitated thoughtful resource allocation and early identification of high-risk patients. However, effective methods to meet these needs are lacking. Objective The aims of this study were to analyze the electronic health records (EHRs) of patients who tested positive for COVID-19 and were admitted to hospitals in the Mount Sinai Health System in New York City; to develop machine learning models for making predictions about the hospital course of the patients over clinically meaningful time horizons based on patient characteristics at admission; and to assess the performance of these models at multiple hospitals and time points. Methods We used Extreme Gradient Boosting (XGBoost) and baseline comparator models to predict in-hospital mortality and critical events at time windows of 3, 5, 7, and 10 days from admission. Our study population included harmonized EHR data from five hospitals in New York City for 4098 COVID-19–positive patients admitted from March 15 to May 22, 2020. The models were first trained on patients from a single hospital (n=1514) before or on May 1, externally validated on patients from four other hospitals (n=2201) before or on May 1, and prospectively validated on all patients after May 1 (n=383). Finally, we established model interpretability to identify and rank variables that drive model predictions. Results Upon cross-validation, the XGBoost classifier outperformed baseline models, with an area under the receiver operating characteristic curve (AUC-ROC) for mortality of 0.89 at 3 days, 0.85 at 5 and 7 days, and 0.84 at 10 days. XGBoost also performed well for critical event prediction, with an AUC-ROC of 0.80 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. In external validation, XGBoost achieved an AUC-ROC of 0.88 at 3 days, 0.86 at 5 days, 0.86 at 7 days, and 0.84 at 10 days for mortality prediction. Similarly, the unimputed XGBoost model achieved an AUC-ROC of 0.78 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. Trends in performance on prospective validation sets were similar. At 7 days, acute kidney injury on admission, elevated LDH, tachypnea, and hyperglycemia were the strongest drivers of critical event prediction, while higher age, anion gap, and C-reactive protein were the strongest drivers of mortality prediction. Conclusions We externally and prospectively trained and validated machine learning models for mortality and critical events for patients with COVID-19 at different time horizons. These models identified at-risk patients and uncovered underlying relationships that predicted outcomes.

show abstract

Tomography and Prognostic Indices in the State of the Art of Evaluation in Hospitalized Patients with COVID-19 Pneumonia

et al. 2022

View full text Add to dashboard Cite

Evaluation in medical emergencies of COVID-19 patients represents a challenge to regulate preventive and timely management. There are key imaging and laboratory tools to classify the severity. The aim of the study was to evaluate the chest CT score performance and prognostic indices in COVID-19 patients to predict the progression to critical illness. This was a retrospective study between run between April and December 2020, in which 109 patients were included. Patients of any age and gender and who required hospitalization due to a confirmed COVID-19 diagnosis by RT-PCR and chest CT and laboratory were analyzed. In 75% of them, there was at least one comorbidity, and 30% developed critical illness, and the average mortality was 10%. In 49.5%, there was a CORADS-5 on admission, and in 50%, there was a peripheral distribution of the interstitial infiltrate in the left lower lobe. The risk factors were FiO2, CT score > 18, and the NRL index. The combination of the high-risk Quick COVID-19 Severity Index (qCSI) plus CT score > 18 indices was the best prediction index for the development of a critical condition. The combined use of indices in infected COVID-19 patients showed diagnostic accuracy and predicted severity. Imaging and the laboratory tests are key tools independent of the wave of recurrence.

show abstract

Development and External Validation of a Prognostic Tool for COVID-19 Critical Disease

Cited by 3 publications

References 25 publications

Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation (Preprint)

Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation (Preprint)

Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation

Tomography and Prognostic Indices in the State of the Art of Evaluation in Hospitalized Patients with COVID-19 Pneumonia

Contact Info

Product

Resources

About