Implementation approaches and barriers for rule-based and machine learning-based sepsis risk prediction tools: a qualitative study

Abstract

Objective Many options are currently available for sepsis surveillance clinical decision support (CDS) from electronic medical record (EMR) vendors, third party, and homegrown models drawing on rule-based (RB) and machine learning (ML) algorithms. This study explores sepsis CDS implementation from the perspective of implementation leads by describing the motivations, tool choices, and implementation experiences of a diverse group of implementers. Materials and Methods Semi-structured interviews were conducted with and a questionnaire was administered to 21 hospital leaders overseeing CDS implementation at 15 US medical centers. Participants were recruited via convenience sampling. Responses were coded by 2 coders with consensus approach and inductively analyzed for themes. Results Use of sepsis CDS is motivated in part by quality metrics for sepsis patients. Choice of tool is driven by ease of integration, customization capability, and perceived predictive potential. Implementation processes for these CDS tools are complex, time-consuming, interdisciplinary undertakings resulting in heterogeneous choice of tools and workflow integration. To improve clinician acceptance, implementers addressed both optimization of the alerts as well as clinician understanding and buy in. More distrust and confusion was reported for ML models, as compared to RB models. Respondents described a variety of approaches to overcome implementation barriers; these approaches related to alert firing, content, integration, and buy-in. Discussion While there are shared socio-technical challenges of implementing CDS for both RB and ML models, attention to user education, support, expectation management, and dissemination of effective practices may improve feasibility and effectiveness of ML models in quality improvement efforts. Conclusion Further implementation science research is needed to determine real world efficacy of these tools. Clinician acceptance is a significant barrier to sepsis CDS implementation. Successful implementation of less clinically intuitive ML models may require additional attention to user confusion and distrust.

A review of risk concepts and models for predicting the risk of primary stroke

“…Even though a more complicated model might have better predictive power, than for example a logistic regression model, if it is a black box model it might not allow for this confidence. While explainable and interpretable AI have been a recent focus within the machine learning field, there is still evidence to show that clinicians and patients distrust in machine learning methods prevents greater uptake (Elish, 2018 ; Mpanya et al, 2021 ; Joshi et al, 2022 ).…”

Influential factors in the adoption of clinical decision support systems in hospital settings: a systematic review and meta-synthesis of qualitative studies

“…Table 1 summarizes the studies included in the synthetic analysis. All the studies were conducted in the context of developing countries with the majority conducted in the UK (Bailey et al, 2020;Cresswell et al, 2015Cresswell et al, , 2017Lee et al, 2016;Mozaffar et al, 2016a, b) and the USA (Ash et al, 2012(Ash et al, , 2015Joshi et al, 2022;Simon et al, 2013) as the frontrunners in healthcare digitisation leaving the developing countries context under-investigated. All but three studies investigated the adoption of commercial CDSS (Ash et al, 2012;Bailey et al, 2020;Joshi et al, 2022) in hospitals, leaving the home-grown context under-investigated.…”

“…All the studies were conducted in the context of developing countries with the majority conducted in the UK (Bailey et al, 2020;Cresswell et al, 2015Cresswell et al, , 2017Lee et al, 2016;Mozaffar et al, 2016a, b) and the USA (Ash et al, 2012(Ash et al, , 2015Joshi et al, 2022;Simon et al, 2013) as the frontrunners in healthcare digitisation leaving the developing countries context under-investigated. All but three studies investigated the adoption of commercial CDSS (Ash et al, 2012;Bailey et al, 2020;Joshi et al, 2022) in hospitals, leaving the home-grown context under-investigated. With respect to study participants, only three studies (Ash et al, 2015;Lee et al, 2016;Mozaffar et al, 2016a) took internal and external (users, managers, IT staff and vendor experts) stakeholders' perceptions into perspective and only one study investigated vendor experts' perception with regard to CDSS adoption (Cresswell et al, 2015).…”

“…There was a lack of consistency with regard to studies' foci so only six studies solely focused on procurement and adoption as the purpose of the study (Baysari et al, 2023;Cresswell et al, 2015Cresswell et al, , 2017Joshi et al, 2022;Lee et al, 2016;Liberati et al, 2017) while the rest mainly focused on development, implementation and utilisation of CDSS. Technology context.…”

“…So there must have been something that is recommending them. (Baysari et al, 2023)(p.3) The cost of establishing CDSS was noted as an on-going challenge (Joshi et al, 2022). It includes the procurement of the hardware, and the software, as well as the necessary infrastructure and initial and ongoing training for users.…”