Deep Reinforcement Learning for Sepsis Treatment

Raghu, Aniruddh; Komorowski, Matthieu; Ahmed, Imran; Celi, Leo Anthony; Szolovits, Peter; Ghassemi, Marzyeh

doi:10.48550/arxiv.1711.09602

Cited by 31 publications

(63 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Consistent with previous work [13], [14], [15], we focus on vasopressor and IV fluid treatments, which whilst regarded important, studies have shown significant variation among patient responses to them [3], [4] [16]. Further there is little agreement among medical researchers on best practices that guide fluid or vasopressor administration beyond initial resuscitation.…”

Section: Introductionmentioning

confidence: 74%

“…This approach was extended by the authors to alternative reward schemes more in line with current medical decision making, and continuous state spaces. Raghu et al [14] used DRL (with a Dueling DDQN [30] algorithm) and a continuous state representation. Peng et al [15] considers partial observability.…”

Section: Related Workmentioning

confidence: 99%

“…Actions: To ensure each action has a considerable representation in the dataset, we discretize vasopressor and fluid administrations into 3 bins, instead of 5 as in previous work [14], [13] [15]. Where 0 indicates no treatment.…”

Section: Pomdp Formulationmentioning

confidence: 99%

See 2 more Smart Citations

Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment

Nanayakkara,

Clermont,

Langmead

et al. 2021

Preprint

View full text Add to dashboard Cite

Sepsis is the leading cause of mortality in the ICU, responsible for 6% of all hospitalizations and 35% of all in-hospital deaths in USA [1]. However, there is no universally agreed upon strategy for vasopressor and fluid administration [2]. It has also been observed that different patients respond differently to treatment [3], [4], highlighting the need for individualized treatment. Vasopressors and fluids are administrated with specific effects to cardiovascular physiology in mind and medical research has suggested that physiologic, hemodynamically guided, approaches to treatment is prudent [5]. Thus we propose a novel approach, exploiting and unifying complementary strengths of Mathematical Modelling, Deep Learning, Reinforcement Learning and Uncertainty Quantification, to learn individualized, safe, and uncertainty aware treatment strategies. We first infer patient-specific, dynamic cardiovascular states using a novel physiology-driven recurrent neural network trained in an unsupervised manner. This information, along with a learned lowdimensional representation of the patient's lab history and observable data, is then used to derive value distributions using Batch Distributional Reinforcement Learning. Moreover in a safety critical domain it is essential to know what our agent does and does not know, for this we also quantify the model uncertainty associated with each patient state and action, and propose a general framework for uncertainty aware, interpretable treatment policies. This framework can be tweaked easily, to reflect a clinician's own confidence of the framework, and can be easily modified to factor in human expert opinion, whenever it's accessible. Using representative patients and a validation cohort, we show that our method has learned physiologically interpretable generalizable policies. We posit that this work takes an important step towards a safe, physiology and uncertainty aware, decision support system for septic patients. It also provides a general framework for using Reinforcement Learning to support healthcare applications involving sequential decisions.

show abstract

Section: Introductionmentioning

confidence: 74%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment

Nanayakkara,

Clermont,

Langmead

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…So, its recommendations can be reviewed or changed by the decision-maker before they are applied. Similar methodologies were applied to sepsis treatment [19], [20]. Offline reinforcement learning can also be applied in many other sequential decision-making problems as in healthcare, spoken dialogue systems, self-driving cars, and robotics [15].…”

Section: A Related Workmentioning

confidence: 99%

“…Off-policy policy evaluation tackles the performance prediction problem producing an estimate that minimizes some concept of error [16]. Alternatives to OPE are, for example, crowd-sourced human labeling agent actions [37], an expert qualitative analysis [20] and policy ranking [27].…”

Section: Off-policy Policy Evaluation (Ope)mentioning

confidence: 99%

Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning

de Lima,

Krohling

2021

Preprint

View full text Add to dashboard Cite

High dropout rates in tertiary education expose a lack of efficiency that causes frustration of expectations and financial waste. Predicting students at risk is not enough to avoid student dropout. Usually, an appropriate aid action must be discovered and applied in the proper time for each student. To tackle this sequential decision-making problem, we propose a decision support method to the selection of aid actions for students using offline reinforcement learning to support decision-makers effectively avoid student dropout. Additionally, a discretization of student's state space applying two different clustering methods is evaluated. Our experiments using logged data of real students shows, through off-policy evaluation, that the method should achieve roughly 1.0 to 1.5 times as much cumulative reward as the logged policy. So, it is feasible to help decision-makers apply appropriate aid actions and, possibly, reduce student dropout.

show abstract

Estimating treatment effects for time-to-treatment antibiotic stewardship in sepsis

et al. 2023

View full text Add to dashboard Cite

Deep Reinforcement Learning for Sepsis Treatment

Cited by 31 publications

References 0 publications

Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment

Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment

Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning

Estimating treatment effects for time-to-treatment antibiotic stewardship in sepsis

Contact Info

Product

Resources

About