Patient-Specific Sedation Management via Deep Reinforcement Learning

Abstract

Introduction: Developing reliable medication dosing guidelines is challenging because individual dose–response relationships are mitigated by both static (e. g., demographic) and dynamic factors (e.g., kidney function). In recent years, several data-driven medication dosing models have been proposed for sedatives, but these approaches have been limited in their ability to assess interindividual differences and compute individualized doses.Objective: The primary objective of this study is to develop an individualized framework for sedative–hypnotics dosing.Method: Using publicly available data (1,757 patients) from the MIMIC IV intensive care unit database, we developed a sedation management agent using deep reinforcement learning. More specifically, we modeled the sedative dosing problem as a Markov Decision Process and developed an RL agent based on a deep deterministic policy gradient approach with a prioritized experience replay buffer to find the optimal policy. We assessed our method's ability to jointly learn an optimal personalized policy for propofol and fentanyl, which are among commonly prescribed sedative–hypnotics for intensive care unit sedation. We compared our model's medication performance against the recorded behavior of clinicians on unseen data.Results: Experimental results demonstrate that our proposed model would assist clinicians in making the right decision based on patients' evolving clinical phenotype. The RL agent was 8% better at managing sedation and 26% better at managing mean arterial compared to the clinicians' policy; a two-sample t-test validated that these performance improvements were statistically significant (p < 0.05).Conclusion: The results validate that our model had better performance in maintaining control variables within their target range, thereby jointly maintaining patients' health conditions and managing their sedation.

Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness Assessment

“…An initial 129 articles were screened on title and abstract, 27 of which were reviewed in full text resulting in 8 included journal articles (19)(20)(21)(22)(23)(24)(25)(26) and 7 conference papers (27)(28)(29)(30)(31)(32)(33). Backward and forward citation chases of the initially included articles returned 904 articles which were screened on title and abstract, of which 90 articles were reviewed in full text resulting in 10 included journal articles (34)(35)(36)(37)(38)(39)(40)(41)(42)(43) and 11 conference papers (44)(45)(46)(47)(48)(49)(50)(51)(52)(53)(54).…”

“…The number of features that were used for the state-space varied from 9 to 63 (median 41,5, 25th percentile = 19,25, 75th percentile = 47), the number of discrete binned states ranged from 5 to 750 states (median 350, 25th percentile = 6,5, 75th percentile = 750). Also, nine articles used a vectorized continuous state-space or availed of different techniques such as neural networks to process this input (Appendix 4, http://links.lww.com/CCM/H448) (19,23,24,26,35,36,(40)(41)(42). The distribution of features that were included in the state space is displayed in Figure 2.…”

Causal inference using observational intensive care unit data: a systematic review and recommendations for future practice

“…Nosocomial infections 8 ( 22) 0 (0) 0 (0) (13) Anti-inflammatory drugs 5 (14) 0 (0) 1 ( 5) (10) Sedatives & analgesics 1 (3) 0 (0) 5 ( 25) (10) Vasopressors & intra-venous fluids 0 (0) 0 (0) 5 (25) (8) Antimicrobials 4 (11) 0 (0) 0 (0) (7) Mechanical ventilation 0 (0) 1 ( 25) 2 (10) (5) Anticoagulants 1 ( 3) 0 (0) 2 (10) (5) Diuretics 3 (8) 0 (0) 0 (0) (5) Renal replacement therapy 3 (8) 0 (0) 0 (0) (5) Other 11 (31) 3 ( 75) 5 ( 25) (32)…”

“…Mortality 25 (69) 1 ( 25) 50) 0 (0) (5) Need for mechanical ventilation 2 ( 6) 0 (0) 0 (0) (3) Other 8 (22) 1 ( 25) 2 ( 10) ( 18) Number of included ICUs 1 14 (39) 2 ( 50) 13 (81) (52) 2-4 5 ( 14) 0 (0) 1 ( 6) (11) 5-10 4 (11) 0 (0) 0 (0) (7) 11-20 6 (17) 1 ( 25) 0 (0) (12) 21-100 2 ( 6) 1 ( 25) 0 (0) (5) >100 5 (14) 0 (0) 2 ( 12) ( 12) Utilised open source databases MIMIC-II 0 (0) 0 (0) 1 (5) (2) MIMIC- III 6 (17) 0 (0) 13 (65…”

“…12 However, ICU physicians are typically confronted with 3 treatment decisions which occur at multiple time-points, ie, time-varying treatments (figure 1, panel 1). 10,11 Estimating the effect of time-varying treatments using observational data is often complicated by treatment-confounder feedback, 13 leading to 'treatment-affected time-varying confounding' (TTC, panel 1) 11,14,15 . Usage of standard methods in the presence of TTC leads to bias.…”