Model enhanced reinforcement learning to enable precision dosing: A theoretical case study with dosing of propofol

Abstract

Extending the potential of precision dosing requires evaluating methodologies offering more flexibility and higher degree of personalization. Reinforcement learning (RL) holds promise in its ability to integrate multidimensional data in an adaptive process built toward efficient decision making centered on sustainable value creation. For general anesthesia in intensive care units, RL is applied and automatically adjusts dosing through monitoring of patient's consciousness. We further explore the problem of optimal control of anesthesia with propofol by combining RL with state‐of‐the‐art tools used to inform dosing in drug development. In particular, we used pharmacokinetic‐pharmacodynamic (PK‐PD) modeling as a simulation engine to generate experience from dosing scenarios, which cannot be tested experimentally. Through simulations, we show that, when learning from retrospective trial data, more than 100 patients are needed to reach an accuracy within the range of what is achieved with a standard dosing solution. However, embedding a model of drug effect within the RL algorithm improves accuracy by reducing errors to target by 90% through learning to take dosing actions maximizing long‐term benefit. Data residual variability impacts accuracy while the algorithm efficiently coped with up to 50% interindividual variability in the PK and 25% in the PD model's parameters. We illustrate how extending the state definition of the RL agent with meaningful variables is key to achieve high accuracy of optimal dosing policy. These results suggest that RL constitutes an attractive approach for precision dosing when rich data are available or when complemented with synthetic data from model‐based tools used in model‐informed drug development.

Reinforcement Learning and PK‐PD Models Integration to Personalize the Adaptive Dosing Protocol of Erdafitinib in Patients with Metastatic Urothelial Carcinoma

“…The proposed RL-approach was developed with this aim. Indeed, differently from the previous works 26,28,30 where the objective was the identification of an optimal dosing strategy for an entire patient population, here, the scope of the RL-approach was to individually optimize the erdafitinib adaptive dosing protocol by tailoring the dose adjustment rules on each patient. In the RL-context, this change of perspective means to move from a unique RL-agent, trained on an entire patient population, to a set of personal QL-agents, each trained on a single individual patient.…”

“…To address this issue, a number of recent works has promoted the idea of coupling RL algorithms with population PK-PD modeling. 21,23,25,26 In this context, PK-PD models were used to simulate virtual patients and, thus, to generate experience from different dosing scenarios on which the RL algorithm could learn the optimal dosing strategy. In almost all the works reported in the literature, a unique RLagent was trained to find an optimal dosing strategy for an entire patient population.…”

Reinforcement learning as an innovative model-based approach: Examples from precision dosing, digital health and computational psychiatry

“…Optimal dosing of propofol administration (Ribba et al, 2022) Just-in-time-adaptive-intervention for HeartSteps, mobile app aimed at reducing physical inactivity (Liao et al, 2020) Population analysis of signal-detection task in anhedonic subjects (Huys et al, 2013) between digital health applications and pharmacological drugs represents a ground for attempting to reframe PMX-a recognized key player in the development of the latter-as a key support to the development of the former, in particular when it comes to precision dosing for digital health.…”

“…We have recently evaluated the performance of RL algorithms for precision dosing of propofol for general anesthesia and for which a meta-analysis showed that the monitoring of the bispectral index (BIS)-a PD endpoint-contributes to reduce the amount of propofol given and the incidence of adverse reactions (Wang et al, 2021). In (Ribba et al, 2022), we performed a theoretical analysis of propofol precision dosing confronting RL to hallmarks of clinical pharmacology problems during drug development, i.e. the low number of patients and tested dosing regimen, the incomplete understanding of the drivers of response and the presence of high variability in the data.…”

Artificial Intelligence: From Buzzword to Useful Tool in Clinical Pharmacology

“…13 Extension of such application for patient care include exploration of novel AI/ML method, such as reinforcement learning for precision dosing in individual patients. 14 One of the most exciting prospects lies in using AI/ML in model-informed drug development. The potential for high impact in this area is immense and is currently an active domain of research.…”