Assessment of Machine Learning–Based Medical Directives to Expedite Care in Pediatric Emergency Medicine

Abstract

IMPORTANCEIncreased wait times and long lengths of stay in emergency departments (EDs) are associated with poor patient outcomes. Systems to improve ED efficiency would be useful. Specifically, minimizing the time to diagnosis by developing novel workflows that expedite test ordering can help accelerate clinical decision-making. OBJECTIVE To explore the use of machine learning-based medical directives (MLMDs) to automate diagnostic testing at triage for patients with common pediatric ED diagnoses. DESIGN, SETTING, AND PARTICIPANTS Machine learning models trained on retrospective electronic health record data were evaluated in a decision analytical model study conducted at the ED of the Hospital for Sick Children Toronto, Canada. Data were collected on all patients aged 0 to 18 years presenting to the ED from July 1, 2018, to June 30, 2019 (77 219 total patient visits). EXPOSURE Machine learning models were trained to predict the need for urinary dipstick testing, electrocardiogram, abdominal ultrasonography, testicular ultrasonography, bilirubin level testing, and forearm radiographs. MAIN OUTCOMES AND MEASURES Models were evaluated using area under the receiver operator curve, true-positive rate, false-positive rate, and positive predictive values. Model decision thresholds were determined to limit the total number of false-positive results and achieve high positive predictive values. The time difference between patient triage completion and test ordering was assessed for each use of MLMD. Error rates were analyzed to assess model bias. In addition, model explainability was determined using Shapley Additive Explanations values. RESULTS There was a total of 42 238 boys (54.7%) included in model development; mean (SD) age of the children was 5.4 (4.8) years. Models obtained high area under the receiver operator curve (0.89-0.99) and positive predictive values (0.77-0.94) across each of the use cases. The proposed implementation of MLMDs would streamline care for 22.3% of all patient visits and make test results available earlier by 165 minutes (weighted mean) per affected patient. Model explainability for each MLMD demonstrated clinically relevant features having the most influence on model predictions.Models also performed with minimal to no sex bias. CONCLUSIONS AND RELEVANCEThe findings of this study suggest the potential for clinical automation using MLMDs. When integrated into clinical workflows, MLMDs may have the potential to autonomously order common ED tests early in a patient's visit with explainability provided to patients and clinicians.

An external validation study of the Score for Emergency Risk Prediction (SERP), an interpretable machine learning-based triage score for the emergency department

“…Brajer et al suggested a machine-learning model fact sheet reporting for end-users 18 . Visualization-based efforts such as population, patient, and temporal level feature importance, or nomograms, could be adopted 19 – 22 . Like the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) or The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines for reporting machine learning results 23 , 24 , there should be a guideline for the standardization of user interfaces (UIs) and a format for clinical decision support for end-users, including clinicians and patients 25 , 26 .…”

“…Brajer et al suggested a machine-learning model fact sheet reporting for end-users 18 . Visualization-based efforts such as population, patient, and temporal level feature importance, or nomograms, could be adopted [19][20][21][22]…”

An external validation study of the Score for Emergency Risk Prediction (SERP), an interpretable machine learning-based triage score for the emergency department

“…Brajer et al suggested a machine-learning model fact sheet reporting for end-users 18 . Visualization-based efforts such as population, patient, and temporal level feature importance, or nomograms, could be adopted [19][20][21][22] . Like the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) or The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines for reporting machine learning results 23,24 , there should be a guideline for the standardization of user interfaces (UIs) and a format for clinical decision support for end-users, including clinicians and patients.…”

Artificial intelligence-based clinical decision support in pediatrics

“… Emergency department testing: Singh et al reported an ML model to predict the need for frequently performed clinical testing (urinary dipstick testing, electrocardiogram, abdominal ultrasonography, testicular ultrasonography, bilirubin level testing, and forearm radiographs) among children presenting to a pediatric emergency department. 34 Using an outcome of testing ordered in triage, the model demonstrated high area under the receiver operating characteristic curve (AUROC) (0.89–0.99, across individual use cases for clinical tests including urinary dipstick testing and electrocardiograms) with a high positive predictive value (0.77–0.94) with results available by a mean time of 165 min. The investigators were further able to characterize model explainability using Shapley Additive Explanation to ensure their clinical relevance.…”