Reinforcement learning assisted oxygen therapy for COVID-19 patients under intensive care

Abstract

Background Patients with severe Coronavirus disease 19 (COVID-19) typically require supplemental oxygen as an essential treatment. We developed a machine learning algorithm, based on deep Reinforcement Learning (RL), for continuous management of oxygen flow rate for critically ill patients under intensive care, which can identify the optimal personalized oxygen flow rate with strong potentials to reduce mortality rate relative to the current clinical practice. Methods We modeled the oxygen flow trajectory of COVID-19 patients and their health outcomes as a Markov decision process. Based on individual patient characteristics and health status, an optimal oxygen control policy is learned by using deep deterministic policy gradient (DDPG) and real-time recommends the oxygen flow rate to reduce the mortality rate. We assessed the performance of proposed methods through cross validation by using a retrospective cohort of 1372 critically ill patients with COVID-19 from New York University Langone Health ambulatory care with electronic health records from April 2020 to January 2021. Results The mean mortality rate under the RL algorithm is lower than the standard of care by 2.57% (95% CI: 2.08–3.06) reduction (P < 0.001) from 7.94% under the standard of care to 5.37% under our proposed algorithm. The averaged recommended oxygen flow rate is 1.28 L/min (95% CI: 1.14–1.42) lower than the rate delivered to patients. Thus, the RL algorithm could potentially lead to better intensive care treatment that can reduce the mortality rate, while saving the oxygen scarce resources. It can reduce the oxygen shortage issue and improve public health during the COVID-19 pandemic. Conclusions A personalized reinforcement learning oxygen flow control algorithm for COVID-19 patients under intensive care showed a substantial reduction in 7-day mortality rate as compared to the standard of care. In the overall cross validation cohort independent of the training data, mortality was lowest in patients for whom intensivists’ actual flow rate matched the RL decisions.

Causal inference using observational intensive care unit data: a systematic review and recommendations for future practice

“…Various methods exist to find optimal DTRs, either from a set of pre-specified regimes or directly from data (for an overview, we refer to the book by Chakraborty and Moodie 60 ). Among the included studies in this review, for example, Shahn and colleagues 31 used 'artificial censoring/IPW' [60][61][62] to estimate the optimal fluid-limiting treatment regime for sepsis patients among a pre-specified set of DTRs (ie, 'fluid caps'). Wang and colleagues 33 used 16 the parametric G formula to estimate the per-protocol (PP) effect of 'low tidal volume ventilation', a pre-specified DTR that was compared with standard care in an earlier RCT.…”

Reinforcement learning evaluation of treatment policies for patients with hepatitis C virus

“…Reinforcement learning is an area of machine learning that studies how actions are taken over time affect current and downstream outcomes [8][9][10][11][12][13][14]18]. An example is a physician who is considering different treatment options for a specific condition, which may require updating based on a patient's response and any adverse events.…”

“…Herein, we used hepatitis C virus (HCV) treatment with direct-acting antivirals (DAA) as a case study on which to develop a reinforcement learning approach to evaluate proposed treatment policies before implementation using historical data. Reinforcement learning provides a framework to utilize data that is longitudinal in nature and contains feedback from decisions made over time, such as assessment of a patient's health status and decision to start treatment and thus evaluate new treatment strategies or policies in medicine [8][9][10][11][12][13][14]. HCV is a valuable case study as it has traditionally been one of the most common risk factors for cirrhosis and liver-related mortality in the United States and Europe.…”

A Systematic Review on Medical Oxygen Ecosystem: Current State and Recent Advancements

“…Automation of oxygen parameter monitoring in patients is another area of optimization that Zheng et al explored [57]. They proposed an automated system based on Reinforcement Learning (RL) to control the flow of oxygen to the patients.…”