Transatlantic transferability of a new reinforcement learning model for optimizing haemodynamic treatment for critically ill patients with sepsis

Roggeveen, Luca F.; Hassouni, Ali el; Ahrendt, Jonas; Guo, Tingjie; Fleuren, Lucas M.; Thoral, Patrick; Girbes, Armand R.J.; Hoogendoorn, Mark; Elbers, Paul

doi:10.1016/j.artmed.2020.102003

Cited by 23 publications

(20 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…G methods and RL methods are often perceived as separate disciplines, but show great similarities. For example, Q-learning 54 (an RL method, used by many of the included studies 46,47,[55][56][57] ) is very similar -and under certain conditions even algebraically equivalent-to G estimation (a G method). 58 An important difference is that G methods are used for modelling both STRs and DTRs, while RL methods typically deal with DTRs.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Causal inference using observational intensive care unit data: a systematic review and recommendations for future practice

Smit

Krijthe

Bommel

et al. 2022

Preprint

View full text Add to dashboard Cite

Aim: To review and appraise the quality of studies that present models for causal inference of time-varying treatment effects in the adult intensive care unit (ICU) and give recommendations to improve future research practice. Data sources: Embase, MEDLINE ALL, Web of Science Core Collection, Google Scholar, medRxiv, and bioRxiv up to March 2, 2022. Study selection: Studies that present models for causal inference that deal with time-varying treatments in adult ICU patients. Data extraction: From the included studies, data was extracted about the study setting and applied methodology. Quality of reporting (QOR) of target trial components and causal assumptions (ie, conditional exchangeability, positivity and consistency) were assessed. Results: 1,714 titles were screened and 60 studies were included, of which 36 (60%) were published in the last 5 years. G methods were the most commonly used (n=40/60, 67%), further divided into inverse-probability-of-treatment weighting (n=36/40, 90%) and the parametric G formula (n=4/40, 10%). The remaining studies (n=20/60, 33%) used reinforcement learning methods. Overall, most studies (n=36/60, 60%) considered static treatment regimes. Only ten (17%) studies fully reported all five target trial components (ie, eligibility criteria, treatment strategies, follow-up period, outcome and analysis plan). The treatment strategies and analysis plan components were not (fully) reported in 38% and 48% of the studies, respectively. The causal assumptions (ie, conditional exchangeability, positivity and consistency) remained unmentioned in 35%, 68% and 88% of the studies, respectively. All three causal assumptions were mentioned (or a check for potential violations was reported) in only six (10%) studies. Sixteen studies (27%) estimated the treatment effect both by adjusting for baseline confounding and by adjusting for baseline and treatment-affected time-varying confounding, which often led to substantial changes in treatment effect estimates. Conclusions: Studies that present models for causal inference in the ICU were found to have incomplete or missing reporting of target trial components and causal assumptions. To achieve actionable artificial intelligence in the ICU, we advocate careful consideration of the causal question of interest, the use of target trial emulation, usage of appropriate causal inference methods and acknowledgement (and ideally examination of potential violations) of the causal assumptions. Systematic review registration: PROSPERO (CRD42022324014)

show abstract

Section: Discussionmentioning

confidence: 99%

“…All five target trial components were fully reported in only ten (17%) studies. 31,[39][40][41][42][43][44][45][46][47] The reporting of the target trial components grouped by used CI method are summarized in figures S3-5 and tabulated for each individual study in tables S4-S6.…”

Section: Target Trial Componentsmentioning

confidence: 99%

Causal inference using observational intensive care unit data: a systematic review and recommendations for future practice

Smit

Krijthe

Bommel

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…With regard to future directions in this field, there are several trends that might be expected: Further maturation of the described systems as well as introduction of new ones; Increased adoption of closed-loop controlled fluid administration. The first scenario, where these systems can be safely operated, will probably be the operating room, where constant supervision by an anesthesiologist provides an important safety net; Deeper understanding of fluid dynamics and their translation to ever-more-complex computational models, meant for better accuracy and validity of both controllers and in silico testing platforms [ 18 , 79 , 80 ]; Introduction of new modalities of artificial intelligence, such as reinforcement learning [ 81 , 82 ] and other deep-learning modalities. While there’s increasing use of deep-learning for anesthesia and critical care-related applications [ 83 , 84 ], we have not identified detailed reports on deep-learning-based systems matching our inclusion criteria, meaning we have not identified a system that incorporates deep-learning-based capabilities into a CL system (or, for that matter, a DS system with a feedback loop of repeat evaluations); Continuing formation of a regulatory pipeline dedicated to autonomous and semi-autonomous controlled systems; Increased use of non-invasive sensors in closed-loop fluid administration systems, as their reliability will gradually increase [ 43 , 85 ], as well as artificial intelligence-based advanced sensing modalities (specifically, feature extraction), such as arterial waveform feature analysis [ 77 , 78 ], aimed at providing personalized resuscitation goals; Gradual increase in the degree of automation—from a regulatory standpoint, decision support systems are generally considered safer and easier to approve.…”

Section: Future Directionsmentioning

confidence: 99%

“…Introduction of new modalities of artificial intelligence, such as reinforcement learning [81,82] and other deep-learning modalities. While there's increasing use of deeplearning for anesthesia and critical care-related applications [83,84], we have not identified detailed reports on deep-learning-based systems matching our inclusion criteria, meaning we have not identified a system that incorporates deep-learning-based capabilities into a CL system (or, for that matter, a DS system with a feedback loop of repeat evaluations); • Continuing formation of a regulatory pipeline dedicated to autonomous and semiautonomous controlled systems;…”

mentioning

confidence: 99%

Closed-Loop Controlled Fluid Administration Systems: A Comprehensive Scoping Review

Avital

Snider

Berard

et al. 2022

JPM

View full text Add to dashboard Cite

Physiological Closed-Loop Controlled systems continue to take a growing part in clinical practice, offering possibilities of providing more accurate, goal-directed care while reducing clinicians’ cognitive and task load. These systems also provide a standardized approach for the clinical management of the patient, leading to a reduction in care variability across multiple dimensions. For fluid management and administration, the advantages of closed-loop technology are clear, especially in conditions that require precise care to improve outcomes, such as peri-operative care, trauma, and acute burn care. Controller design varies from simplistic to complex designs, based on detailed physiological models and adaptive properties that account for inter-patient and intra-patient variability; their maturity level ranges from theoretical models tested in silico to commercially available, FDA-approved products. This comprehensive scoping review was conducted in order to assess the current technological landscape of this field, describe the systems currently available or under development, and suggest further advancements that may unfold in the coming years. Ten distinct systems were identified and discussed.

show abstract

“…Similarly, [19] used RL to develop policies for individualized treatment strategies to correct hypotension in Sepsis. More recent work showed that these developed policies for optimizing hemodynamic treatment for critically ill patients with Sepsis are transferable across different patient populations [18]. Furthermore, this work proposes an in-depth inspection approach for clinical interpretability.…”

Section: Related Workmentioning

confidence: 99%

pH-RL: A Personalization Architecture to Bring Reinforcement Learning to Health Practice

Hassouni¹,

Hoogendoorn²,

Čihařová³

et al. 2022

Machine Learning, Optimization, and Data Science

Self Cite

View full text Add to dashboard Cite

While reinforcement learning (RL) has proven to be the approach of choice for tackling many complex problems, it remains challenging to develop and deploy RL agents in real-life scenarios successfully. This paper presents pH-RL (personalization in e-Health with RL), a general RL architecture for personalization to bring RL to health practice. pH-RL allows for various levels of personalization in health applications and allows for online and batch learning. Furthermore, we provide a general-purpose implementation framework that can be integrated with various healthcare applications. We describe a step-by-step guideline for the successful deployment of RL policies in a mobile application. We implemented our open-source RL architecture and integrated it with the MoodBuster mobile application for mental health to provide messages to increase daily adherence to the online therapeutic modules. We then performed a comprehensive study with human participants over a sustained period. Our experimental results show that the developed policies learn to select appropriate actions consistently using only a few days' worth of data. Furthermore, we empirically demonstrate the stability of the learned policies during the study.

show abstract

Transatlantic transferability of a new reinforcement learning model for optimizing haemodynamic treatment for critically ill patients with sepsis

Cited by 23 publications

References 29 publications

Causal inference using observational intensive care unit data: a systematic review and recommendations for future practice

Causal inference using observational intensive care unit data: a systematic review and recommendations for future practice

Closed-Loop Controlled Fluid Administration Systems: A Comprehensive Scoping Review

pH-RL: A Personalization Architecture to Bring Reinforcement Learning to Health Practice

Contact Info

Product

Resources

About