The Epic Sepsis Model Falls Short—The Importance of External Validation

Habib, Anand R.; Lin, Anthony; Grant, Richard W.

doi:10.1001/jamainternmed.2021.3333

Cited by 32 publications

(18 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the argument can be made that further evaluation is needed and that variability in results may be related to variability between hospitals and differential workflow creation. 33 One study, particularly highlighting significant change management effort and delivery of alerts via a mobile application in addition to implementation of an algorithm, did demonstrate 53% decrease in mortality. 13 Another study, conducted at 21 hospitals involving 374 838 patients, demonstrated lower mortality, a lower incidence of ICU admission, and a shorter length of hospital stay using a full rapid response intervention program built around a validated deterioration prediction model.…”

Section: Discussionmentioning

confidence: 99%

Implementation approaches and barriers for rule-based and machine learning-based sepsis risk prediction tools: a qualitative study

et al. 2022

View full text Add to dashboard Cite

Objective Many options are currently available for sepsis surveillance clinical decision support (CDS) from electronic medical record (EMR) vendors, third party, and homegrown models drawing on rule-based (RB) and machine learning (ML) algorithms. This study explores sepsis CDS implementation from the perspective of implementation leads by describing the motivations, tool choices, and implementation experiences of a diverse group of implementers. Materials and Methods Semi-structured interviews were conducted with and a questionnaire was administered to 21 hospital leaders overseeing CDS implementation at 15 US medical centers. Participants were recruited via convenience sampling. Responses were coded by 2 coders with consensus approach and inductively analyzed for themes. Results Use of sepsis CDS is motivated in part by quality metrics for sepsis patients. Choice of tool is driven by ease of integration, customization capability, and perceived predictive potential. Implementation processes for these CDS tools are complex, time-consuming, interdisciplinary undertakings resulting in heterogeneous choice of tools and workflow integration. To improve clinician acceptance, implementers addressed both optimization of the alerts as well as clinician understanding and buy in. More distrust and confusion was reported for ML models, as compared to RB models. Respondents described a variety of approaches to overcome implementation barriers; these approaches related to alert firing, content, integration, and buy-in. Discussion While there are shared socio-technical challenges of implementing CDS for both RB and ML models, attention to user education, support, expectation management, and dissemination of effective practices may improve feasibility and effectiveness of ML models in quality improvement efforts. Conclusion Further implementation science research is needed to determine real world efficacy of these tools. Clinician acceptance is a significant barrier to sepsis CDS implementation. Successful implementation of less clinically intuitive ML models may require additional attention to user confusion and distrust.

show abstract

Section: Discussionmentioning

confidence: 99%

Implementation approaches and barriers for rule-based and machine learning-based sepsis risk prediction tools: a qualitative study

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Recent findings in clinical practice strongly reinforce this learning. [ 13,56 ] Unlike the situation for pharmaceutical drugs, in vitro diagnostics (IVD), and medical devices, many digital medicine technologies, particularly in the mobile applications and even software‐as‐medical device (SaMD) segments (classified in the USA as medical device software (MDSW) under the EU's Medical Device Regulation (MDR)) do not have sufficient clinical evidence for efficacy and safety to make clinicians and healthcare organizations confident in implementing them in clinical practice. Evidence is often only developed for technical validity, with insufficient proof of validated usability and only rarely is clinical and health economic evidence of benefit generated.…”

Section: Lack Of Robust Clinical Evidence Basementioning

confidence: 99%

Overcoming Pilotitis in Digital Medicine at the Intersection of Data, Clinical Evidence, and Adoption

Egermark

Blasiak

Remus

et al. 2022

Advanced Intelligent Systems

View full text Add to dashboard Cite

The number of digital medicine technologies, a category of tools that use software and algorithmically driven products to measure or intervene to improve human health, [1] has expanded enormously in recent years, further accelerated by the need for remote care delivery during the COVID-19 pandemic, and now ranges from telemedicine to serious gaming, wearable devices, AI-supported diagnostics, and clinical decision support systems (CDSSs), among others. Building on the experiences during the pandemic, there is a real potential for the digital transformation of health care to the benefit of patients worldwide. However, the promising developments to date are at risk of stalling due to inherent limitations in the way digital medicine and especially medical artificial intelligence (AI) technologies are being developed and implemented today. [2] Already in 2017, the number of digital medicine technologies on the worldwide market exceeded 318 000, with the mobile application segment growing by over 200 daily. [3] There are widespread expectations that the surge will lead to improvement of health care globally, [4] not least evidenced by the WHO declaration on digital health, [5] and by the rapid growth of private and public investment in this field. [6,7] The advent of digital medicine bears special promise for low-and middleincome countries (LMIC), where applications such as telemedicine and CDSS may enable these countries to "leapfrog" from today's often insufficient healthcare systems into more efficient, accessible, and patient-centric models, [8] similar to what has been seen in Africa's finance sector. [9] The level of improvement possible would correspond to the digital transformation seen in other sectors of society, such as media and retail industries, with the digitalization of processes and service models, beyond the mere digitization of traditional healthcare practices seen so

show abstract

“…The model was later found to have a lower AUC upon external validation than initially reported. 21 Due to the cutoffs used, it was eventually only found to catch 7% of sepsis patients missed by a physician and missed sepsis in 67% of sepsis patients despite generating alerts on 18% of all hospitalized patients. 21 This further underscores the importance of rigorously testing and scrutinizing a model before deployment, as well as having the input of physicians to make sure that the tool is clinically meaningful.…”

Section: Changing Disease Rates In Populations May Render Amentioning

confidence: 99%

Machine learning and medicine- a brief introduction

Lee¹,

Peterson²

2022

The Chronicles

View full text Add to dashboard Cite

Artificial Intelligence (AI) and Machine Learning (ML) have advanced rapidly in recent years and now have the potential to change medicine. This review serves as an introduction to AI and the potential it has to change medicine. Specific examples of past milestones particularly in the domain of critical care are presented, including algorithms that have been developed that can interpret chest x-rays or predict clinical outcomes such as extubation failure or ICU mortality. Included is a brief general discussion of what AI is, how it is made, and how physicians will be involved with it. Arguments are then presented as to why AI will likely not leave physicians without a job, including expectations vs. reality, AI still requires human supervision, new discoveries bring new challenges, and AI cannot design itself. Far from displacing physicians, AI, if implemented well, stands poised to automate repetitive tasks, make physicians more accurate, and free them up to spend more time with patients.

show abstract

The Epic Sepsis Model Falls Short—The Importance of External Validation

Cited by 32 publications

References 5 publications

Implementation approaches and barriers for rule-based and machine learning-based sepsis risk prediction tools: a qualitative study

Implementation approaches and barriers for rule-based and machine learning-based sepsis risk prediction tools: a qualitative study

Overcoming Pilotitis in Digital Medicine at the Intersection of Data, Clinical Evidence, and Adoption

Machine learning and medicine- a brief introduction

Contact Info

Product

Resources

About