Deep Inverse Reinforcement Learning for Sepsis Treatment

Yu, Chao; Ren, Guoqi; Liu, Jiming

doi:10.1109/ichi.2019.8904645

Cited by 32 publications

(23 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[4] From the experience pool, a batch of N tuples will be selected to learn the optimal policy. [5] The temporal difference loss function is computed. [6] The critic network is updated by minimizing the temporal difference loss.…”

Section: Training Validation and Testing Set Partitionmentioning

confidence: 99%

“…In the past few decades, several RL-based models have been proposed to regulate sedation in the ICU (6,7,(21)(22)(23)(24)(25)(26)(27)(28)(29). However, most sedation management methods exhibit one or more of the following limitations: (1) incomplete physiological context or patient response variability, (2) use of simulated data for validation, (3) failure to account for common clinical practices such as attempts to minimize the total dosage of sedatives (17), and ( 4) assumption of discrete state and action spaces resulting in sensitivities to heuristic choices of discretization levels (5). Lastly, most of the prior work has focused on a specific medication-propofol-which has no intrinsic analgesic effect and must be coadministered with an opioid or other analgesic for ICU patients (30).…”

Section: Introductionmentioning

confidence: 99%

“…Interest in the applications of RL to healthcare has grown steadily over the last decade. Within the last few years, numerous works have demonstrated the potential of RL methods to help manage sensitive treatment decisions in sepsis ( 1 – 5 ), sedation regulation ( 6 , 7 ), mechanical ventilation ( 1 , 8 ), and medication dosing ( 9 – 11 ). Refer to the works of Liu and Prescott ( 12 ) and Yu et al ( 1 ) for a recent systematic review of RL models in critical care and healthcare.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Patient-Specific Sedation Management via Deep Reinforcement Learning

Eghbali

Alhanai

Ghassemi

2021

Front. Digit. Health

View full text Add to dashboard Cite

Introduction: Developing reliable medication dosing guidelines is challenging because individual dose–response relationships are mitigated by both static (e. g., demographic) and dynamic factors (e.g., kidney function). In recent years, several data-driven medication dosing models have been proposed for sedatives, but these approaches have been limited in their ability to assess interindividual differences and compute individualized doses.Objective: The primary objective of this study is to develop an individualized framework for sedative–hypnotics dosing.Method: Using publicly available data (1,757 patients) from the MIMIC IV intensive care unit database, we developed a sedation management agent using deep reinforcement learning. More specifically, we modeled the sedative dosing problem as a Markov Decision Process and developed an RL agent based on a deep deterministic policy gradient approach with a prioritized experience replay buffer to find the optimal policy. We assessed our method's ability to jointly learn an optimal personalized policy for propofol and fentanyl, which are among commonly prescribed sedative–hypnotics for intensive care unit sedation. We compared our model's medication performance against the recorded behavior of clinicians on unseen data.Results: Experimental results demonstrate that our proposed model would assist clinicians in making the right decision based on patients' evolving clinical phenotype. The RL agent was 8% better at managing sedation and 26% better at managing mean arterial compared to the clinicians' policy; a two-sample t-test validated that these performance improvements were statistically significant (p < 0.05).Conclusion: The results validate that our model had better performance in maintaining control variables within their target range, thereby jointly maintaining patients' health conditions and managing their sedation.

show abstract

Section: Training Validation and Testing Set Partitionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Patient-Specific Sedation Management via Deep Reinforcement Learning

Eghbali

Alhanai

Ghassemi

2021

Front. Digit. Health

View full text Add to dashboard Cite

show abstract

“…RL aims to solve this kind of sequential decisionmaking problems when an agent chooses an action at each time step based on its current state, and receives an evaluative feedback and the new state from the environment [1]. In the past decades, applying RL for more efficient decision-making has become a hot research topic in healthcare domains [2], generating a great breakthrough in treatment of diabetics [3], cancer [4], sepsis [5], and many other diseases [6][7][8].…”

Section: Introductionmentioning

confidence: 99%

Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units

Ren

Dong

2020

BMC Med Inform Decis Mak

Self Cite

View full text Add to dashboard Cite

Background: Reinforcement learning (RL) provides a promising technique to solve complex sequential decision making problems in healthcare domains. Recent years have seen a great progress of applying RL in addressing decision-making problems in Intensive Care Units (ICUs). However, since the goal of traditional RL algorithms is to maximize a long-term reward function, exploration in the learning process may have a fatal impact on the patient. As such, a short-term goal should also be considered to keep the patient stable during the treating process. Methods: We use a Supervised-Actor-Critic (SAC) RL algorithm to address this problem by combining the long-term goal-oriented characteristics of RL with the short-term goal of supervised learning. We evaluate the differences between SAC and traditional Actor-Critic (AC) algorithms in addressing the decision making problems of ventilation and sedative dosing in ICUs. Results: Results show that SAC is much more efficient than the traditional AC algorithm in terms of convergence rate and data utilization. Conclusions: The SAC algorithm not only aims to cure patients in the long term, but also reduces the degree of deviation from the strategy applied by clinical doctors and thus improves the therapeutic effect.

show abstract

“…While there does not exist an intuitive method for risk stratification of patients with penile cancer, there is strong evidence of the potential of machine learning for stratification of patients in other diseases such as heart failure [7][8][9][10], kidney disease [11], and critical care [12][13][14][15][16][17]. Furthermore, machine learning has been shown to be effective for readmission prediction [17][18][19], drug adverse event prediction [20].…”

Section: Introductionmentioning

confidence: 99%

Machine learning algorithm for early mortality prediction in patients with advanced penile cancer

Chen

Kudelka

Rosado

et al. 2020

Preprint

View full text Add to dashboard Cite

Penile cancer remains a rare cancer with an annual incidence of 1 in 100,000 men in the United States, accounting for 0.4-0.6% of all malignancies. Furthermore, to date there are no predictive models of early mortality in penile cancer. Meanwhile, machine learning has potential to serve as a prognostic tool for patients with advanced disease. We developed a machine learning model for predicting early mortality in penile cancer (survival less than 11 months after initial diagnosis. A cohort of 88 patients with advanced penile cancer was extracted from the Surveillance, Epidemiology and End Results (SEER) program. In the cohort, patients with advanced penile cancer exhibited a median overall survival of 21 months, with the 25th percentila of overall survival being 11 months. We constructed predictive features based on patient demographics, staging, metastasis, lymph node biopsy criteria, and metastatic sites. We trained a multivariate logistic regression model, tuning parameters with respect to regularization, and feature selection criteria. Upon evaluation with 5-fold cross validation, our model achieved 68.2% accuracy with AUC 0.696. Criteria for advanced staging (T4, group stage IV), as well as higher age, white race and squamous cell histology, were the most predictive of early mortality. Tumor size was the strongest negative predictor of early mortality. Our study showcases the first known predictive model for early mortality in patients with advanced penile cancer and should serve as a framework for approaching the clinical problem in future studies. Future work should aim to incorporate other data sources such as genomic and metabolomic data, increase patient counts, incorporate clinical characteristics such as ECOG and RECIST criteria, and assess the performance of the model in a prospective fashion.

show abstract

Deep Inverse Reinforcement Learning for Sepsis Treatment

Cited by 32 publications

References 6 publications

Patient-Specific Sedation Management via Deep Reinforcement Learning

Patient-Specific Sedation Management via Deep Reinforcement Learning

Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units

Machine learning algorithm for early mortality prediction in patients with advanced penile cancer

Contact Info

Product

Resources

About