Effectiveness of an Artificial Intelligence–Enabled Intervention for Detecting Clinical Deterioration

Abstract

ImportanceInpatient clinical deterioration is associated with substantial morbidity and mortality but may be easily missed by clinicians. Early warning scores have been developed to alert clinicians to patients at high risk of clinical deterioration, but there is limited evidence for their effectiveness.ObjectiveTo evaluate the effectiveness of an artificial intelligence deterioration model–enabled intervention to reduce the risk of escalations in care among hospitalized patients using a study design that facilitates stronger causal inference.Design, Setting, and ParticipantsThis cohort study used a regression discontinuity design that controlled for confounding and was based on Epic Deterioration Index (EDI; Epic Systems Corporation) prediction model scores. Compared with other observational research, the regression discontinuity design facilitates causal analysis. Hospitalized adults were included from 4 general internal medicine units in 1 academic hospital from January 17, 2021, through November 16, 2022.ExposureAn artificial intelligence deterioration model–enabled intervention, consisting of alerts based on an EDI score threshold with an associated collaborative workflow among nurses and physicians.Main Outcomes and MeasuresThe primary outcome was escalations in care, including rapid response team activation, transfer to the intensive care unit, or cardiopulmonary arrest during hospitalization.ResultsDuring the study, 9938 patients were admitted to 1 of the 4 units, with 963 patients (median [IQR] age, 76.1 [64.2-86.2] years; 498 males [52.3%]) included within the primary regression discontinuity analysis. The median (IQR) Elixhauser Comorbidity Index score in the primary analysis cohort was 10 (0-24). The intervention was associated with a −10.4–percentage point (95% CI, −20.1 to −0.8 percentage points; P = .03) absolute risk reduction in the primary outcome for patients at the EDI score threshold. There was no evidence of a discontinuity in measured confounders at the EDI score threshold.Conclusions and RelevanceUsing a regression discontinuity design, this cohort study found that the implementation of an artificial intelligence deterioration model–enabled intervention was associated with a significantly decreased risk of escalations in care among inpatients. These results provide evidence for the effectiveness of this intervention and support its further expansion and testing in other care settings.

Clinical Evaluations of Early Warning Scores—Reply

“…In Reply We appreciate Goulden’s interest in our study and the nuance raised regarding the interpretation of regression discontinuity designs. In the article, we showed sensitivity analyses for the choice of bandwidths ranging from 1 to 15 points on either side of the threshold; however, not all bandwidth choices may be relevant given that larger bandwidths include patients farther away from the threshold who are less exchangeable.…”

“…Larger bandwidths include patients farther from the threshold that may introduce bias even after adjusting for the running variable. In particular, we would expect increased bias with increasing bandwidths beyond risk scores of 72 considering that we observed significant delays between first score of 65 or greater and maximum score at bandwidths greater than 72 (eFigure 7 in Supplement 1) . There are bandwidth selection procedures that choose a data-driven balance between bias and precision, such as the method used in our study.…”

Clinical Evaluations of Early Warning Scores

“…To the Editor I have 2 concerns regarding the article by Gallo et al evaluating the effectiveness of an electronic health record (EHR)-based early warning score. First, a visual inspection of eFigure 2 in Supplement 2 (the association between the running variable and the primary outcome) does not show an obvious discontinuity in the outcome; it shows how sensitive the outcome result is to the bandwidth selection, with implausibly large effects at small bandwidth sizes, and no significant effect at bandwidth sizes of 9 or above.…”

“…Second, the study’s primary outcome was a composite of rapid response team (RRT) activation, intensive care unit (ICU) transfer, and cardiopulmonary arrest, which they collectively referred to as “escalation of care” and justified as being the outcomes that were “most preventable with an early warning system.” Their secondary outcome was the same composite with the addition of inpatient death . The implication is that inpatient death is not readily preventable with an early warning system, even though death is one of the outcomes that the Epic Deterioration Index (Epic Systems Corporation) score is supposed to predict.…”

“…The implication is that inpatient death is not readily preventable with an early warning system, even though death is one of the outcomes that the Epic Deterioration Index (Epic Systems Corporation) score is supposed to predict. Of note, the primary outcome was statistically significant (−8.3; 95% CI, −15.7 to −0.8; P = .03), whereas the slightly different secondary outcome was insignificant . It would be useful to know whether the choice of primary outcome was prespecified.…”