External Validation and Comparison of a General Ward Deterioration Index Between Diversely Different Health Systems

Abstract

OBJECTIVES: Implementing a predictive analytic model in a new clinical environment is fraught with challenges. Dataset shifts such as differences in clinical practice, new data acquisition devices, or changes in the electronic health record (EHR) implementation mean that the input data seen by a model can differ significantly from the data it was trained on. Validating models at multiple institutions is therefore critical. Here, using retrospective data, we demonstrate how Predicting Intensive Care Transfers and other UnfoReseen Events (PICTURE), a deterioration index developed at a single academic medical center, generalizes to a second institution with significantly different patient population. DESIGN: PICTURE is a deterioration index designed for the general ward, which uses structured EHR data such as laboratory values and vital signs. SETTING: The general wards of two large hospitals, one an academic medical center and the other a community hospital. SUBJECTS: The model has previously been trained and validated on a cohort of 165,018 general ward encounters from a large academic medical center. Here, we apply this model to 11,083 encounters from a separate community hospital. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: The hospitals were found to have significant differences in missingness rates (> 5% difference in 9/52 features), deterioration rate (4.5% vs 2.5%), and racial makeup (20% non-White vs 49% non-White). Despite these differences, PICTURE’s performance was consistent (area under the receiver operating characteristic curve [AUROC], 0.870; 95% CI, 0.861–0.878), area under the precision-recall curve (AUPRC, 0.298; 95% CI, 0.275–0.320) at the first hospital; AUROC 0.875 (0.851–0.902), AUPRC 0.339 (0.281–0.398) at the second. AUPRC was standardized to a 2.5% event rate. PICTURE also outperformed both the Epic Deterioration Index and the National Early Warning Score at both institutions. CONCLUSIONS: Important differences were observed between the two institutions, including data availability and demographic makeup. PICTURE was able to identify general ward patients at risk of deterioration at both hospitals with consistent performance (AUROC and AUPRC) and compared favorably to existing metrics.

Validation of a Proprietary Deterioration Index Model and Performance in Hospitalized Adults

“…In late 2022, Epic switched to an observation-level evaluation method because it “more accurately represents the experience clinicians have when using the model.” Epic currently reports a mean AUROC of 0.710 (used with permission), which lies between our calculated observation- and encounter-level AUROCs. Compared with the largest existing external validation of the DTI, our observation-level AUROC was similar (0.759 vs 0.768-0.780), while our encounter-level AUROC was substantially lower (0.685 vs 0.821-0.863). We found a general pattern of decreasing performance with increasing lead times for mechanical ventilation and ICU transfer, whereas performance for death remained relatively high and stable across lead times.…”

“…Compared with the largest existing external validation of the DTI, 20 our observation-level AUROC was similar (0.759 vs 0.768-0.780), while our encounter-level AUROC was substantially lower (0.685 vs 0.821-0.863). We found a general pattern of decreasing performance with increasing lead times for mechanical ventilation and ICU transfer, whereas performance for death remained relatively high and stable across lead times.…”

“…We used both observation- and encounter-level evaluation methods for our primary analysis, consistent with existing evaluation studies of deterioration prediction models 16 , 17 , 18 , 19 , 20 (eMethods in Supplement 1 ). At the observation level, we included all DTI scores across each hospitalization and asked whether the outcome occurred within 12 hours of each prediction.…”

Use of a Continuous Single Lead Electrocardiogram Analytic to Predict Patient Deterioration Requiring Rapid Response Team Activation

“…AHI-PI may also compliment EHR based early warning systems which in combination could increase accuracy, provide greater insight into the underlying cause of the alert, and assist in more rapid adjudication of the alarms. [33][34][35] Furthermore, since AHI-PI is approved for use with various single lead wearable ECG patches, it may allow more intensive and informative monitoring of floor level patients who are monitored much less frequently.…”

Novel machine learning model to improve performance of an early warning system in hospitalized patients: a retrospective multisite cross-validation study

“…A DI score value ranges between 0 and 100, defining low (<30 green), intermediate (30–60 orange), or high risk (>60 red) of a composite AE: all-cause mortality, cardiac arrest, transfer to intensive care, and evaluation by the rapid response team. 4 , 5 , 6 Each generated risk score is determined based on routinely recorded physiological, clinical, and laboratory parameters within Epic’s EHR to support medical decision-making. The risk score is determined based on age, neurological assessment, cardiac rhythm, oxygen requirement, Glasgow Coma Scale, vital sign measurements (temperature, systolic blood pressure, pulse rate, oxygen saturation, respiratory rate), and laboratory values (hematocrit, white blood cell count, blood urea nitrogen, potassium, sodium, blood pH, platelet count).…”