Improving and Externally Validating Mortality Prediction Models for COVID-19 Using Publicly Available Data

Abstract

We conducted a systematic survey of COVID-19 endpoint prediction literature to: (a) identify publications that include data that adhere to FAIR (findability, accessibility, interoperability, and reusability) principles and (b) develop and reuse mortality prediction models that best generalize to these datasets. The largest such cohort data we knew of was used for model development. The associated published prediction model was subjected to recursive feature elimination to find a minimal logistic regression model which had statistically and clinically indistinguishable predictive performance. This model could still not be applied to the four external validation sets that were identified, due to complete absence of needed model features in some external sets. Thus, a generalizable model (GM) was built which could be applied to all four external validation sets. An age-only model was used as a benchmark, as it is the simplest, effective, and robust predictor of mortality currently known in COVID-19 literature. While the GM surpassed the age-only model in three external cohorts, for the fourth external cohort, there was no statistically significant difference. This study underscores: (1) the paucity of FAIR data being shared by researchers despite the glut of COVID-19 prediction models and (2) the difficulty of creating any model that consistently outperforms an age-only model due to the cohort diversity of available datasets.

A robust COVID-19 mortality prediction calculator based on Lymphocyte count, Urea, C-Reactive Protein, Age and Sex (LUCAS) with chest X-rays

“…The ability to include CXR results is not widely available in other prediction calculators and has been included in a study 35 along with ten other parameters (symptoms, past medical history and measurables). More recently, some of the studies have included the CXR imaging in prognostic models 45 , 46 , with good accuracy; however, they have either utilised information such as electronic health records 45 including comorbidities 46 , 47 , which are not always known at the point of care, additional blood biomarkers such as D-Dimer 7 , 41 and lactate dehydrogenase 42 , which are not measured routinely during triage, or incorporated complex deep-learning methodologies 46 , affecting the explainability and simplicity of the model. Indeed, in a parallel study, we have developed a highly accurate deep-learning based model (DenResCov-19) to classify from CXR images patients positive for SARS-CoV-2, tuberculosis, and other forms of pneumonia 6 , which will be integrated into the LUCAS calculator in a future study.…”

Using artificial intelligence and predictive modelling to enable learning healthcare systems (LHS) for pandemic preparedness

“…The creation, dissemination, and application of evidence-based knowledge are critical in enabling effective and timely responses to disease outbreaks such as COVID-19. AI plays a pivotal role in supporting the knowledge pillar of LHS by unlocking valuable insights from vast amounts of data by enabling rapid analysis through techniques such as ML and NLP [35] , [36] , [37] . These techniques were instrumental in the accurate identification of COVID-19 cases and informing treatment decisions during COVID-19.…”

COVID-19 machine learning model predicts outcomes in older patients from various European countries, between pandemic waves, and in a cohort of Asian, African, and American patients

“…The European cohort included 2,858 patients with an average mortality rate of 45% both in ICU and 30 days after ICU admission, while 13% of patients were at low risk of deterioration. European patients' age median was 75 years (IQR, [72-78]), with 30% female, and median length of ICU stay was 13 days (IQR, [6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22]). The distribution of patients among the European countries, including the number of patients as well as ICU mortality rate per country is shown in France was chosen as the validation cohort to assess the generalisability among the European cohort because it had the highest number of patients in the database (647, or 22% of the European cohort), with 40% mortality rate (ICU and 30-day) and 19% of patients with a low risk of deterioration.…”

“…However, current efforts have been limited by lack of generalisation to diverse patient populations, between patients admitted at different waves and small sample sizes. While there are many studies investigating prediction of outcomes in COVID-19 patients [10][11][12][13][14], only a handful have investigated generalisability of the models across countries with diverse populations located in different continents using imaging [15] and no studies have been found that used routinely collected data, as outlined in [15]. Indeed, a recent review on chest imaging, emphasised the importance of validation dataset to assess generalisability of the model to other cohorts, rather than only on the sampled population [16].…”