Machine Learning Versus Usual Care for Diagnostic and Prognostic Prediction in the Emergency Department: A Systematic Review

Abstract

Objective Having shown promise in other medical fields, we sought to determine whether machine learning (ML) models perform better than usual care in diagnostic and prognostic prediction for emergency department (ED) patients. Methods In this systematic review, we searched MEDLINE, Embase, Central, and CINAHL from inception to October 17, 2019. We included studies comparing diagnostic and prognostic prediction of ED patients by ML models to usual care methods (triage‐based scores, clinical prediction tools, clinician judgment) using predictor variables readily available to ED clinicians. We extracted commonly reported performance metrics of model discrimination and classification. We used the PROBAST tool for risk of bias assessment (PROSPERO registration: CRD42020158129). Results The search yielded 1,656 unique records, of which 23 studies involving 16,274,647 patients were included. In all seven diagnostic studies, ML models outperformed usual care in all performance metrics. In six studies assessing in‐hospital mortality, the best‐performing ML models had better discrimination (area under the receiver operating characteristic curve [AUROC] =0.74–0.94) than any clinical decision tool (AUROC =0.68–0.81). In four studies assessing hospitalization, ML models had better discrimination (AUROC =0.80–0.83) than triage‐based scores (AUROC =0.68–0.82). Clinical heterogeneity precluded meta‐analysis. Most studies had high risk of bias due to lack of external validation, low event rates, and insufficient reporting of calibration. Conclusions Our review suggests that ML may have better prediction performance than usual care for ED patients with a variety of clinical presentations and outcomes. However, prediction model reporting guidelines should be followed to provide clinically applicable data. Interventional trials are needed to assess the impact of ML models on patient‐centered outcomes.

Models Predicting Hospital Admission of Adult Patients Utilizing Prehospital Data: Systematic Review Using PROBAST and CHARMS

“…Potential sources of bias that may cause flawed or distorted model predictions were found in every model, for example, from minor (not reporting handling of missing values [ 38 , 39 , 43 , 44 , 47 ], univariate predictor selection [ 39 , 47 ]) to potentially damaging (dichotomized continuous variables [ 22 , 41 , 43 ], low events-per-variable [ 44 ], no external validation [ 38 , 41 , 42 , 44 , 47 ]), which suggest that study reports of models’ abilities to predict outcomes have the potential to be flawed. This is consistent with other evaluations of prediction modeling studies [ 34 ], including evaluations applying CHARMS and PROBAST in the emergency department setting [ 35 , 36 ].…”

“…As the number of prediction modeling publications continue to grow, the need exists to apply the same rigor to systematic reviews of health care–related prediction modeling as that which has been applied to clinical trial and other types of systematic reviews through the use of tools, such as PROBAST and CHARMS, to facilitate quality assessment for individual prediction model studies using standardized guidelines [ 30 , 33 ]. Only two systematic reviews [ 35 , 36 ] that have focused on increasing overall throughput by decreasing emergency department boarding and systemic exit block in health systems applied the rigorous PROBAST and CHARMS methodologies, with both reporting a high degree of bias in the studies that they examined.…”

Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review

“…A systematic review of 23 studies about machine learning for diagnostic and prognostic predictions in emergency departments found that analysis was the most poorly rated domain, with 20 studies at high risk of bias. 32 This study found deficiencies in how continuous variables and missing data were handled, and that model calibration was rarely reported. Another publication about machine learning risk prediction models for triage of patients in the emergency department also considered 22/25 studies at high risk of bias.…”

Machine learning techniques for mortality prediction in emergency departments: a systematic review

“…However, the authors mentioned that many studies have limited applicability to clinical practice and there are other considerations more than performance metrics as well as barriers that should be taken into account to have successful ML models in real life. 51 Therefore, despite the great research successes in building ML-based predictive models for clinical practice, there remain few examples of ML models being successfully integrated into the daily routine or critical parts of clinical environments. 52 This reveals that what is being done in research is not completely in line with the realities of clinical practice.…”