Predicting COVID-19 prognosis in the ICU remained challenging: external validation in a multinational regional cohort

Meijs, Daniek A. M.; Kuijk, Sander M. J. van; Wynants, Laure; Stessel, Björn; Mehagnoul-Schipper, Jannet; Hana, Anisa; Scheeren, Clarissa I.E.; Bergmans, Dennis C. J. J.; Bickenbach, Johannes; Laenen, Margot Vander; Smits, Luc; Horst, Iwan C C van der; Marx, Gernot; Mesotten, Dieter; Bussel, Bas C.T. van; Heijnen, Nanon F. L.; Mulder, Marlies; Koelmann, Marcel; Bels, Julia L.M.; Wilmes, Nick; Hendriks, Charlotte W.E.; Janssen, Emma B. N. J.; Florack, Micheline C. D. M.; Ghossein‐Doha, Chahinda; Woude, Meta C. E. van der; Bormans-Russell, Laura; Pierlet, Noёlla; Goethuys, Ben; Bruggen, J. A. R. Van; Vermeiren, G.; Vervloessem, Hendrik; Boer, Willem

doi:10.1016/j.jclinepi.2022.10.015

Cited by 7 publications

(8 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our findings are in keeping with those of Meijs et al ( 18 ) who used data from a regional clinical network to validate COVID-19 models in ICU, although these were developed on general wards and tested in ICUs. Even in 2020, the low quality, high volume of published COVID-19 models was recognized as a significant problem with calls for better data sharing and better reporting ( 48 ).…”

Section: Discussionsupporting

confidence: 91%

See 1 more Smart Citation

External Validation of Prognostic Models in Critical Care: A Cautionary Tale From COVID-19 Pneumonitis

Bate,

Stokes,

Greenlee

et al. 2024

Critical Care Explorations

View full text Add to dashboard Cite

OBJECTIVES (BACKGROUND): To externally validate clinical prediction models that aim to predict progression to invasive ventilation or death on the ICU in patients admitted with confirmed COVID-19 pneumonitis. DESIGN: Single-center retrospective external validation study. DATA SOURCES: Routinely collected healthcare data in the ICU electronic patient record. Curated data recorded for each ICU admission for the purposes of the U.K. Intensive Care National Audit and Research Centre (ICNARC). SETTING: The ICU at Manchester Royal Infirmary, Manchester, United Kingdom. PATIENTS: Three hundred forty-nine patients admitted to ICU with confirmed COVID-19 Pneumonitis, older than 18 years, from March 1, 2020, to February 28, 2022. Three hundred two met the inclusion criteria for at least one model. Fifty-five of the 349 patients were admitted before the widespread adoption of dexamethasone for the treatment of severe COVID-19 (pre-dexamethasone patients). OUTCOMES: Ability to be externally validated, discriminate, and calibrate. METHODS: Articles meeting the inclusion criteria were identified, and those that gave sufficient details on predictors used and methods to generate predictions were tested in our cohort of patients, which matched the original publications’ inclusion/exclusion criteria and endpoint. RESULTS: Thirteen clinical prediction articles were identified. There was insufficient information available to validate models in five of the articles; a further three contained predictors that were not routinely measured in our ICU cohort and were not validated; three had performance that was substantially lower than previously published (range C-statistic = 0.483–0.605 in pre-dexamethasone patients and C = 0.494–0.564 among all patients). One model retained its discriminative ability in our cohort compared with previously published results (C = 0.672 and 0.686), and one retained performance among pre-dexamethasone patients but was poor in all patients (C = 0.793 and 0.596). One model could be calibrated but with poor performance. CONCLUSIONS: Our findings, albeit from a single center, suggest that the published performance of COVID-19 prediction models may not be replicated when translated to other institutions. In light of this, we would encourage bedside intensivists to reflect on the role of clinical prediction models in their own clinical decision-making.

show abstract

Section: Discussionsupporting

confidence: 91%

“…We anticipated these changes may have impacted the performance of models that aim to predict survival for patients with COVID-19, particularly in an ICU setting. There have been external validations of COVID-19 prediction models, but there is yet to be one that specifically focuses on models developed in ICUs rather than general wards (3,(15)(16)(17)(18).…”

mentioning

confidence: 99%

External Validation of Prognostic Models in Critical Care: A Cautionary Tale From COVID-19 Pneumonitis

Bate,

Stokes,

Greenlee

et al. 2024

Critical Care Explorations

View full text Add to dashboard Cite

show abstract

“…The GDPR has some stricter rules compared to HIPAA, making it unattractive for US instances to incorporate EU data in their public data repositories. Consequently, EU citizens are insufficiently represented in research on open data 27 . Therefore, we argue that European institutes should openly share patient data, such that European patients are incorporated in developing novel health technologies and can benefit from them accordingly.…”

Section: Discussionmentioning

confidence: 99%

A guide to sharing open healthcare data under the General Data Protection Regulation

de Kok,

de la Hoz,

de Jong

et al. 2023

Sci Data

View full text Add to dashboard Cite

Sharing healthcare data is increasingly essential for developing data-driven improvements in patient care at the Intensive Care Unit (ICU). However, it is also very challenging under the strict privacy legislation of the European Union (EU). Therefore, we explored four successful open ICU healthcare databases to determine how open healthcare data can be shared appropriately in the EU. A questionnaire was constructed based on the Delphi method. Then, follow-up questions were discussed with experts from the four databases. These experts encountered similar challenges and regarded ethical and legal aspects to be the most challenging. Based on the approaches of the databases, expert opinion, and literature research, we outline four distinct approaches to openly sharing healthcare data, each with varying implications regarding data security, ease of use, sustainability, and implementability. Ultimately, we formulate seven recommendations for sharing open healthcare data to guide future initiatives in sharing open healthcare data to improve patient care and advance healthcare.

show abstract

“…SpO2/FiO2 may substitute for PaO2/FiO2 as a diagnostic and prognostic marker in COVID-19 patients ( 51 ). The performance differences do exist in different race/ethnicity and regions; external validation is needed ( 52 ). Multi-center, cross-country studies to verify the accuracy of these severity scores need to be conducted.…”

Section: Discussionmentioning

confidence: 99%

Outcome prediction in hospitalized COVID-19 patients: Comparison of the performance of five severity scores

Chung

Tang²,

Chen³

et al. 2023

Front. Med.

View full text Add to dashboard Cite

BackgroundThe aim of our study was to externally validate the predictive capability of five developed coronavirus disease 2019 (COVID-19)-specific prognostic tools, including the COVID-19 Spanish Society of Infectious Diseases and Clinical Microbiology (SEIMC), Shang COVID severity score, COVID-intubation risk score-neutrophil/lymphocyte ratio (IRS-NLR), inflammation-based score, and ventilation in COVID estimator (VICE) score.MethodsThe medical records of all patients hospitalized for a laboratory-confirmed COVID-19 diagnosis between May 2021 and June 2021 were retrospectively analyzed. Data were extracted within the first 24 h of admission, and five different scores were calculated. The primary and secondary outcomes were 30-day mortality and mechanical ventilation, respectively.ResultsA total of 285 patients were enrolled in our cohort. Sixty-five patients (22.8%) were intubated with ventilator support, and the 30-day mortality rate was 8.8%. The Shang COVID severity score had the highest numerical area under the receiver operator characteristic (AUC-ROC) (AUC 0.836) curve to predict 30-day mortality, followed by the SEIMC score (AUC 0.807) and VICE score (AUC 0.804). For intubation, both the VICE and COVID-IRS-NLR scores had the highest AUC (AUC 0.82) compared to the inflammation-based score (AUC 0.69). The 30-day mortality increased steadily according to higher Shang COVID severity scores and SEIMC scores. The intubation rate exceeded 50% in the patients stratified by higher VICE scores and COVID-IRS-NLR score quintiles.ConclusionThe discriminative performances of the SEIMC score and Shang COVID severity score are good for predicting the 30-day mortality of hospitalized COVID-19 patients. The COVID-IRS-NLR and VICE showed good performance for predicting invasive mechanical ventilation (IMV).

show abstract

Predicting COVID-19 prognosis in the ICU remained challenging: external validation in a multinational regional cohort

Cited by 7 publications

References 39 publications

External Validation of Prognostic Models in Critical Care: A Cautionary Tale From COVID-19 Pneumonitis

External Validation of Prognostic Models in Critical Care: A Cautionary Tale From COVID-19 Pneumonitis

A guide to sharing open healthcare data under the General Data Protection Regulation

Outcome prediction in hospitalized COVID-19 patients: Comparison of the performance of five severity scores

Contact Info

Product

Resources

About