Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing

Abstract

Machine learning-based clinical decision support tools for sepsis create opportunities to identify at-risk patients and initiate treatments earlier, critical to improving sepsis outcomes. Increasing use of such systems necessitates quantifying and understanding provider adoption. Using realtime provider interactions with a sepsis early detection tool (Targeted Real-time Early Warning System) deployed at five hospitals over a two-year period (469,419 screened encounters, 9,805 (2.1%) retrospectively-identified sepsis cases), we found high sensitivity (82% of sepsis cases identified), high adoption rates (89% of alerts evaluated by a physician or advanced practice provider and 38% of evaluated alerts confirmed) and an association between use of the tool and earlier treatment of sepsis patients (1.85 (95% CI:1.66-2.00) hour reduction in median time to first antibiotics order). Further, we found that provider-related factors were strongly associated with adoption. Beyond improving system performance, efforts to improve adoption should focus on provider knowledge, experience, and perceptions of the system.

Human–machine teaming is key to AI adoption: clinicians’ experiences with a deployed machine learning system

“…Developed from 5 years of historical data collected from three hospitals, TREWS uses an ML approach to learn patterns from time-series data to predict, in real-time, whether a patient is at risk of developing sepsis 21,22,36,37 . In order to account for the heterogeneity of patients with sepsis, the risk prediction method automatically discovers multiple phenotypes of sepsis and learns from provider behavior over time to improve sequential predictions.…”

“…In order to account for the heterogeneity of patients with sepsis, the risk prediction method automatically discovers multiple phenotypes of sepsis and learns from provider behavior over time to improve sequential predictions. It also reduces false positive alerts, thus improving precision, by accounting for confounding comorbidities that can cause automated systems to mistakenly identify a patient as having sepsis 21,38 . Based on provider feedback, the version of TREWS deployed at this hospital waited for an indicator of organ dysfunction prior to alerting.…”

“…While the in-depth interview format limited the number of interviews conducted, we interviewed physicians and nurses across several unit types to get a broad perspective. A follow-up quantitative study looking at 2 years of data from this and four other hospitals using the same ML-based system shows that high adoption is sustained across all sites 21 . However, the high receptivity to the system observed here may not be typical across deployments of new technologies.…”

“…Through a series of in-depth interviews with 20 physicians and nurses, analyzed following a grounded theory approach (see Methods), we explore these issues in the context of a deployed ML-based system: the Targeted Real-time Early Warning System (TREWS) for sepsis, a syndrome with high mortality and morbidity wherein a whole-body inflammatory response causes organ dysfunction [17][18][19][20] . Leveraging an ML-based risk score, TREWS was developed to support clinicians' timely identification and treatment of sepsis (See Methods for system description) 21,22 . Whereas most ML systems have not yet been broadly deployed and adopted in clinical practice [23][24][25] , TREWS was integrated into the electronic health record (EHR) at a large non-teaching hospital beginning in 2017 and was in use for over 6 months at the time of this study, with high, sustained adoption 21 .…”

“…Leveraging an ML-based risk score, TREWS was developed to support clinicians' timely identification and treatment of sepsis (See Methods for system description) 21,22 . Whereas most ML systems have not yet been broadly deployed and adopted in clinical practice [23][24][25] , TREWS was integrated into the electronic health record (EHR) at a large non-teaching hospital beginning in 2017 and was in use for over 6 months at the time of this study, with high, sustained adoption 21 . At a moment when ML-based support is poised for broad deployment across the healthcare system, this work seeks to inform system designers, implementation scientists, clinicians, and healthcare administrators by clarifying the experiences and views of clinicians tasked with using this emerging technology in their day-to-day work.…”