Explainable Machine Learning models for Rapid Risk Stratification in the Emergency Department: A multi-center study

Abstract

IntroductionRisk stratification of patients presenting to the emergency department (ED) is important for appropriate triage. Using machine learning technology, we can integrate laboratory data from a modern emergency department and present these in relation to clinically relevant endpoints for risk stratification. In this study, we developed and evaluated transparent machine learning models in four large hospitals in the Netherlands.MethodsHistorical laboratory data (2013-2018) available within the first two hours after presentation to the ED of Maastricht University Medical Centre+ (Maastricht), Meander Medical Center (Amersfoort), and Zuyderland (locations Sittard and Heerlen) were used. We used the first five years of data to develop the model and the sixth year to evaluate model performance in each hospital separately. Performance was assessed using area under the receiver-operating-characteristic curve (AUROC), brier scores and calibration curves. The SHapley Additive exPlanations (SHAP) algorithm was used to obtain transparent machine learning models.ResultsWe included 266,327 patients with more than 7 million laboratory results available for analysis. Models possessed high diagnostic performance with AUROCs of 0.94 [0.94-0.95], 0.98 [0.97-0.98], 0.88 [0.87-0.89] and 0.90 [0.89-0.91] for Maastricht, Amersfoort, Sittard and Heerlen, respectively. Using the SHAP algorithm, we visualized patient characteristics and laboratory results that drive patient-specific RISKINDEX predictions. As an illustrative example, we applied our models in a triage system for risk stratification that categorized 94.7% of the patients as low risk with a corresponding NPV of ≥99%.DiscussionDeveloped machine learning models are transparent with excellent diagnostic performance in predicting 31-day mortality in ED patients across four hospitals. Follow up studies will assess whether implementation of these algorithm can improve clinically relevant endpoints.

Machine learning for risk stratification in the emergency department (MARS-ED) study protocol for a randomized controlled pilot trial on the implementation of a prediction model based on machine learning technology predicting 31-day mortality in the emergency department

“…age and sex) and the results of laboratory tests available within the first two hours of ED presentation to create an individualized, precise and rapid risk estimation of 31-day mortality. The RISK INDEX has externally been validated in three other hospitals, showing high prognostic accuracy (AUC ranging from 0.88 to 0.98) [16]. To the best of our knowledge, the MARS-ED study is the first prospective, randomized study to investigate the clinical impact of a ML based prediction model in the ED.…”

“…However, the generalizability of the results of our study is enhanced because we are performing a broadly designed trial in which all patients who enter the ED for assessment and treatment by an internal medicine specialist can be included, and in which there is also a delayed consent pathway to include patients who are temporarily unable to provide informed consent. Furthermore, a recent multicenter validation study showed that the RISK INDEX can be adapted to each medical center's population [16]. In that study in four EDs, the RISK INDEX showed very high discriminatory performance (AUC ranging from 0.88 to 0.98), indicating that the RISK INDEX is applicable despite local differences in patient demographics.…”

“…The RISK INDEX was developed using ML technology at the MUMC + and aims to predict all-cause 31-day mortality for adults ED patients [16]. The score is calibrated ranging from 0 to 100, where a high RISK INDEX…”

“…Recently, machine learning (ML) technology has emerged as a promising approach to develop new prediction models [14,15]. One such new prediction model, the RISK INDEX , has been developed at the Maastricht University Medical Center (MUMC+) [16]. The RISK INDEX utilizes sex, age and routine laboratory tests from the ED to predict 31-day mortality.…”

“…The RISK INDEX utilizes sex, age and routine laboratory tests from the ED to predict 31-day mortality. The RISK INDEX was externally validated in three other medical centers demonstrating an AUC of 0.88 to 0.98 16 . Although ML based prediction models show great potential, to our knowledge, randomized clinical trials to investigate their actual clinical impact have rarely been performed.…”