Evaluating Adoption, Impact, and Factors Driving Adoption for TREWS, a Machine Learning-Based Sepsis Alerting System

Abstract

Machine learning-based clinical decision support tools for sepsis create opportunities to identify at-risk patients and initiate treatments earlier, an important step in improving sepsis outcomes. Increasing use of such systems means quantifying and understanding provider adoption is critical. Using real-time provider interactions with a sepsis early detection tool (Targeted Real-time Early Warning System) deployed at five hospitals over a two-year period (469,419 screened patient encounters, 9,805 (2.1%) of which were retrospectively identified as having sepsis), we found high adoption rates (89% of alerts were evaluated by a physician or advanced practice provider) and an association between use of the tool and earlier treatment of sepsis patients (1.85 (95% CI: 1.66 - 2.00) hour reduction in median time to first antibiotics order). Further, we found that provider-related factors had the strongest association with alert adoption and that case complexity and atypical presentation were associated with dismissal of alerts on sepsis patients. Beyond improving the performance of the system, efforts to improve adoption should focus on provider knowledge, experience, and perceptions of the system.

Trust in Clinical AI: Expanding the Unit of Analysis

“…Wang et al found that a lack of training lessened trust in the AI, as clinicians had to learn how to use and understand the system alone [8]. Henry et al discuss how in a sepsis alert system, clinicians might dismiss the alert if there are not clear signs of sepsis and the patient has a less common presentation of sepsis [54]. Training clinicians on how the system can detect this rarity would be crucial [54].…”

“…Henry et al discuss how in a sepsis alert system, clinicians might dismiss the alert if there are not clear signs of sepsis and the patient has a less common presentation of sepsis [54]. Training clinicians on how the system can detect this rarity would be crucial [54]. Training and onboarding are vital forms of trust calibration [36].…”

“…Whether or not a trusted, senior clinician trusts an AI will influence the rest of the team's trust in the AI [91]. Similarly, clinicians may be required to interact with an EHR system more, increasing situational trust [54]. The implementation of AI will have a broader effect on trust between healthcare professionals, and in turn, will affect trust in the AI [121].…”

Beyond prediction: Off‐target uses of artificial intelligence‐based predictive analytics in a learning health system

“…An extreme version of off‐target use is a one‐size‐fits‐all approach, that is, a model trained on a specific event as a general tool for deterioration throughout the hospital. Examples include the Rothman Index 18 (trained on death in the next 12 months), eCART 19 , 20 (trained on cardiac arrest on the wards), and TREWScore 21 (trained on septic shock in the ICU). Note that these are trained models rather than scores such as SIRS, (q)SOFA and (x)EWS that are fashioned by experts.…”