Comparison of Severity of Illness Scores and Artificial Intelligence Models That Are Predictive of Intensive Care Unit Mortality: Meta-analysis and Review of the Literature

Abstract

Background Severity of illness scores—Acute Physiology and Chronic Health Evaluation, Simplified Acute Physiology Score, and Sequential Organ Failure Assessment—are current risk stratification and mortality prediction tools used in intensive care units (ICUs) worldwide. Developers of artificial intelligence or machine learning (ML) models predictive of ICU mortality use the severity of illness scores as a reference point when reporting the performance of these computational constructs. Objective This study aimed to perform a literature review and meta-analysis of articles that compared binary classification ML models with the severity of illness scores that predict ICU mortality and determine which models have superior performance. This review intends to provide actionable guidance to clinicians on the performance and validity of ML models in supporting clinical decision-making compared with the severity of illness score models. Methods Between December 15 and 18, 2020, we conducted a systematic search of PubMed, Scopus, Embase, and IEEE databases and reviewed studies published between 2000 and 2020 that compared the performance of binary ML models predictive of ICU mortality with the performance of severity of illness score models on the same data sets. We assessed the studies' characteristics, synthesized the results, meta-analyzed the discriminative performance of the ML and severity of illness score models, and performed tests of heterogeneity within and among studies. Results We screened 461 abstracts, of which we assessed the full text of 66 (14.3%) articles. We included in the review 20 (4.3%) studies that developed 47 ML models based on 7 types of algorithms and compared them with 3 types of the severity of illness score models. Of the 20 studies, 4 (20%) were found to have a low risk of bias and applicability in model development, 7 (35%) performed external validation, 9 (45%) reported on calibration, 12 (60%) reported on classification measures, and 4 (20%) addressed explainability. The discriminative performance of the ML-based models, which was reported as AUROC, ranged between 0.728 and 0.99 and between 0.58 and 0.86 for the severity of illness score–based models. We noted substantial heterogeneity among the reported models and considerable variation among the AUROC estimates for both ML and severity of illness score model types. Conclusions ML-based models can accurately predict ICU mortality as an alternative to traditional scoring models. Although the range of performance of the ML models is superior to that of the severity of illness score models, the results cannot be generalized due to the high degree of heterogeneity. When presented with the option of choosing between severity of illness score or ML models for decision support, clinicians should select models that have been externally validated, tested in the practice environment, and updated to the patient population and practice environment. Trial Registration PROSPERO CRD42021203871; https://tinyurl.com/28v2nch8

D-Dimer as a Prognostic Factor in a Tertiary Center Intensive Coronary Care Unit

“…As the vast majority of data on diagnostic and prognostic yield of D-dimer to date relates to thromboembolic disease and given the fact that age-adjusted D-dimer cutoff (being 500 ng/ml for ages ≤ 50 years old and age*10 for ages>50 years old) has become consensus in the field of venous thromboembolic disease 1,28,29 we confirmed our study results by analysis of age-adjusted D-dimer cutoff as well.…”

“…Similar with most prior publications, older patients had an increased percent of elevated D-dimer levels. 1,13,16,28,29 Nevertheless, we have also noticed an increased percent of younger (<40 y/o) patients with elevated D-dimer levels. To explore the explanation for this finding we compared the various admission diagnoses between young <40 y/o) and older ICCU patients and found a significantly increased incidence of myocarditis, pulmonary emboli, and CHF among the young patients (43% vs 16%, p 0.02).…”

“…Based on our study results revealing both in-hospital and 1-year post discharge increased mortality among ICCU patients with elevated D-dimer levels, we recommend increasing the monitoring level of these ICCU patients during their admission, as well as rigorous and frequent post-discharge clinical follow-ups, to ensure early diagnosis of developing medical issues and targeted efforts for their early resolution. Importantly, as of today there are numerous models predicting general ICU mortality including APACHE, SAPS II, and othes 29 but few if any such models for ICCU patients. 35 Accordingly, the result of the current study might held develop a contemporary mortality model for ICCU patients as well.…”

Passive digital markers for Alzheimer's disease and other related dementias: A systematic evidence review

“…In conclusion, before using PDM models as decision support systems for early detection of ADRD, clinicians must be mindful of the similarities and discrepancies between the cohort used for model development and the local practice population, the practice setting, the model's ability to function prospectively, and the models' lead times. 48 They should acquire knowledge of the model's performance during testing in the local practice and ensure that it is periodically updated to changes in patient characteristics and clinical variables and adjusted to new clinical practices and therapeutics. Clinicians should confirm that the models' data are monitored and validated, that the model's performance is periodically updated, and that the model makes the correct recommendation for the right reasons.…”

“…Practitioners should identify clinical performance metrics that evaluate the impact of the PDM tool on the quality of care and patient outcomes, and account for variability in practice. 48…”