Assuring the safety of AI-based clinical decision support systems: a case study of the AI Clinician for sepsis treatment

Abstract

ObjectivesEstablishing confidence in the safety of Artificial Intelligence (AI)-based clinical decision support systems is important prior to clinical deployment and regulatory approval for systems with increasing autonomy. Here, we undertook safety assurance of the AI Clinician, a previously published reinforcement learning-based treatment recommendation system for sepsis.MethodsAs part of the safety assurance, we defined four clinical hazards in sepsis resuscitation based on clinical expert opinion and the existing literature. We then identified a set of unsafe scenarios, intended to limit the action space of the AI agent with the goal of reducing the likelihood of hazardous decisions.ResultsUsing a subset of the Medical Information Mart for Intensive Care (MIMIC-III) database, we demonstrated that our previously published ‘AI clinician’ recommended fewer hazardous decisions than human clinicians in three out of our four predefined clinical scenarios, while the difference was not statistically significant in the fourth scenario. Then, we modified the reward function to satisfy our safety constraints and trained a new AI Clinician agent. The retrained model shows enhanced safety, without negatively impacting model performance.DiscussionWhile some contextual patient information absent from the data may have pushed human clinicians to take hazardous actions, the data were curated to limit the impact of this confounder.ConclusionThese advances provide a use case for the systematic safety assurance of AI-based clinical systems towards the generation of explicit safety evidence, which could be replicated for other AI applications or other clinical contexts, and inform medical device regulatory bodies.

Evaluating the Human Safety Net: Observational study of Physician Responses to Unsafe AI Recommendations in high-fidelity Simulation

“…The definition of unsafe recommendations was based on extreme under-or over-dosing of fluid and/or vasopressor as per previous work. 13 All participating physicians were fully debriefed at the conclusion of the study on the synthetic nature of the AI recommendations so as not to bias their opinions of future interactions with AI-driven systems. During each trial, all physician responses were recorded by a member of the research team sitting in a dead angle in the simulation suite.…”

“…10,11 Attempts have been made to improve the safety profile of AI-driven decision support in retrospective intensive care settings. [12][13][14] Still, the necessity of prospective and higher fidelity evaluations involving clinical end users is clear from recent examples in other fields. 13 For instance, an acute kidney injury alert system showing good performance on retrospective data was found to worsen outcomes when deployed in a real-world setting, illustrating the need for a careful transition between retrospective testing and prospective deployment of digital systems.…”

“…[12][13][14] Still, the necessity of prospective and higher fidelity evaluations involving clinical end users is clear from recent examples in other fields. 13 For instance, an acute kidney injury alert system showing good performance on retrospective data was found to worsen outcomes when deployed in a real-world setting, illustrating the need for a careful transition between retrospective testing and prospective deployment of digital systems. 15 Safely transitioning from "bytes to bedside" is a particularly complex challenge because of the dynamic interaction with human users who are prone to biases and can behave in unpredictable ways.…”

Which haemodynamic monitoring should we chose for critically ill patients with acute circulatory failure?

“…Artificial intelligence will certainly help better define therapeutic strategies in the future. Recent studies using basic demographic and haemodynamic data (not provided by PAC or TPTD) already showed the improvement it may provide in the treatment of septic patients [41 ▪▪ ].…”

Generating synthetic data with a mechanism-based Critical Illness Digital Twin: Demonstration for Post Traumatic Acute Respiratory Distress Syndrome

“…new and varied practice sites) available for model training is the most common proposed response to addressing contextual generalizability (an approach that was effectively applied in the TREWS studies), there is no currently accepted method to account for temporal data drift other than retraining. While many have consigned themselves to the perpetual need to retrain these systems (and have developed processes that will help accomplish this while minimizing significant clinical impact) [7], there is an alternative approach drawn from industrial/technological fields that can potentially enhance the efficiency and effectiveness of utilizing existing data to increase generalizable prediction. This approach is the concept of a digital twin , which refers to an individualized computational representation from within a specific class of objects or type of industrial process.…”