Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine

Abstract

Traditionally, precision medicine involves classifying patients to identify subpopulations that respond favorably to specific therapeutics. We pose precision medicine as a dynamic feedback control problem, where treatment administered to a patient is guided by measurements taken during the course of treatment. We consider sepsis, a life-threatening condition in which dysregulation of the immune system causes tissue damage. We leverage an existing simulation of the innate immune response to infection and apply deep reinforcement learning (DRL) to discover an adaptive personalized treatment policy that specifies effective multicytokine therapy to simulated sepsis patients based on systemic measurements. The learned policy achieves a dramatic reduction in mortality rate over a set of 500 simulated patients relative to standalone antibiotic therapy. Advantages of our approach are threefold: (1) the use of simulation allows exploring therapeutic strategies beyond clinical practice and available data, (2) advances in DRL accommodate learning complex therapeutic strategies for complex biological systems, and (3) optimized treatments respond to a patient's individual disease progression over time, therefore, capturing both differences across patients and the inherent randomness of disease progression within a single patient. We hope that this work motivates both considering adaptive personalized multicytokine mediation therapy for sepsis and exploiting simulation with DRL for precision medicine more broadly.

Genetic Algorithms for model refinement and rule discovery in a high-dimensional agent-based model of inflammation

“…In future work, we will utilize this genetically diverse in silico cohort as part of our machine-learning therapeutic discovery workflow (10,30). We note the importance of in silico genetic diversity for therapeutic discovery in (10); in this work, we developed a multi-cytokine/multi-time-point therapeutic regimen which decreased the mortality rate from ~80% to ~20% for a severe simulated injury.…”

Agent‐based models of inflammation in translational systems biology: A decade later

“…In addition to the ability to facilitate the development and use of increasingly sophisticated ABMs, there have also been methodological improvements for both improving ABMs as well as analyzing the output of simulation experiments utilizing them (Figure ). These developments include work on uncertainty quantification in ABMs (Marino, Hogue, Ray, & Kirschner, ), sensitivity analysis in ABMs (Alam et al, ), methods for increasing the computational efficiency of ABMs via “tuneable resolution” (Kirschner, Hunt, Marino, Fallahi‐Sichani, & Linderman, ), the use of Bayesian statistical model checking for parameter estimation in ABMs (Hussain et al, ), the use of optimization algorithms in conjunction with ABMs (Cicchese, Pienaar, Kirschner, & Linderman, ; R. C. Cockrell & An, ), the use of HPC (C. Cockrell & An, ; R. C. Cockrell & An, ; R. C. Cockrell et al, ; Petersen et al, ; Seekhao et al, ), strategies for data‐driven model validation (Renardy et al, ), and the incorporation of model‐based dynamic control discovery (R. C. Cockrell & An, ; Petersen et al, ). These are exciting developments that have, without a doubt, increased the range of biomedical problems and applications to which ABMs could be applied.…”

“…C. Cockrell & An, 2018), the use of HPC (C. Cockrell & An, 2017; R. C. Cockrell & An, 2018;R. C. Cockrell et al, 2015;Petersen et al, 2019;Seekhao et al, 2018), strategies for datadriven model validation (Renardy et al, 2019), and the incorporation of model-based dynamic control discovery (R. C. Cockrell & An, 2018;Petersen et al, 2019). These are exciting developments that have, without a doubt, increased the range of biomedical problems and applications to which ABMs could be applied.…”

Complex Stability and an Irrevertible Transition Reverted by Peptide and Fibroblasts in a Dynamic Model of Innate Immunity

“…Cockrell and An (74) recently also reached the conclusion that multi-dented therapies are needed to deal with sepsis. They built a reinforcement learning workflow for designing corresponding therapies (29), but they do not analyze the source of the complexity in the special type of irreversibility nor do they address the use of anti-FLC peptides.…”

“…The simulations helped explain why so many of these clinical trials have failed (28): inter-patient variability and the role of multiple factors rather than of the single drug target (27). More recently, Petersen et al (29) put in place a deep reinforcement learning strategy, automatically personalizing this type of model for use in multifold cytokine therapies. Mavroudis et al (30) showed how stochasticity can be taken into account, which they did essentially for the simpler four-variable model referred to above.…”