An external validation study of the Score for Emergency Risk Prediction (SERP), an interpretable machine learning-based triage score for the emergency department

Abstract

Emergency departments (EDs) are experiencing complex demands. An ED triage tool, the Score for Emergency Risk Prediction (SERP), was previously developed using an interpretable machine learning framework. It achieved a good performance in the Singapore population. We aimed to externally validate the SERP in a Korean cohort for all ED patients and compare its performance with Korean triage acuity scale (KTAS). This retrospective cohort study included all adult ED patients of Samsung Medical Center from 2016 to 2020. The outcomes were 30-day and in-hospital mortality after the patients’ ED visit. We used the area under the receiver operating characteristic curve (AUROC) to assess the performance of the SERP and other conventional scores, including KTAS. The study population included 285,523 ED visits, of which 53,541 were after the COVID-19 outbreak (2020). The whole cohort, in-hospital, and 30 days mortality rates were 1.60%, and 3.80%. The SERP achieved an AUROC of 0.821 and 0.803, outperforming KTAS of 0.679 and 0.729 for in-hospital and 30-day mortality, respectively. SERP was superior to other scores for in-hospital and 30-day mortality prediction in an external validation cohort. SERP is a generic, intuitive, and effective triage tool to stratify general patients who present to the emergency department.

Prospective and External Validation of Prognostic Machine Learning Models for Short- and Long-Term Mortality Among Acutely Admitted Patients Based on Blood Tests.

“…The predictive model used in this study incorporated various variable, including demographic, diagnosis code, administrative information and Charlson comorbidity Index. Similarly, an ED triage tool, the Score for Emergency Risk Prediction (SERP), to predict mortality within 2 to 30 days for ED patients was initially applied in a cohort from a Singaporean ED and subsequently underwent external validation in a South Korean ED 20,21 . These studies demonstrated AUCs of 0•81 − 0•82 for in-hospital mortality and 0•80 − 0•82 for 30-day mortality prediction.…”

On the computational approaches for supporting triage systems

“…44 In particular, when considering that decisions made by the triage system can result in death (in case of a wrong decision) or the saved life, the legal and ethical implications require that the model be transparent, even to enable medical staff to have counterintuitive choices. We should report that only a few works by Yu et al 43 and Liu et al 28 pay attention to the explainability of their models. Consequently, the development of explainable AI models is a promising area.…”

“…The problem of explainability of ML models has been explicitly faced by Leung et al 30 The authors of this paper proposed an interpretable ML model for predicting hospitalisation. Yu et al, 43 developed an interpretable ML model to calculate a Score for Emergency Risk Prediction for each patient. The score is validated as a predictor of mortality in the ED.…”

A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes

“…Moreover, it enables users to build transparent and interpretable clinical scores quickly in a straightforward manner. It has been extensively used in different clinical applications, e.g., for general risk assessments in the emergency department, 22 , 23 , 35 , 36 and for prediction of disease-specific outcomes in specific patient cohorts. 24 , 25 , 26 , 37 , 38 , 39 , 40 , 41 …”