EpidemiOptim: A Toolbox for the Optimization of Control Policies in Epidemiological Models

Abstract

Modeling the dynamics of epidemics helps to propose control strategies based on pharmaceuticaland non-pharmaceutical interventions (contact limitation, lockdown, vaccination,etc). Hand-designing such strategies is not trivial because of the number of possibleinterventions and the difficulty to predict long-term effects. This task can be cast as an optimization problem where state-of-the-art machine learning methods such as deep reinforcement learning might bring significant value. However, the specificity of each domain|epidemic modeling or solving optimization problems|requires strong collaborationsbetween researchers from different fields of expertise. This is why we introduce EpidemiOptim, a Python toolbox that facilitates collaborations between researchers inepidemiology and optimization. EpidemiOptim turns epidemiological models and cost functions into optimization problems via a standard interface commonly used by optimization practitioners (OpenAI Gym). Reinforcement learning algorithms based on QLearning with deep neural networks (DQN) and evolutionary algorithms (NSGA-II) are already implemented. We illustrate the use of EpidemiOptim to find optimal policies fordynamical on-o lockdown control under the optimization of the death toll and economic recess using a Susceptible-Exposed-Infectious-Removed (SEIR) model for COVID-19. Using EpidemiOptim and its interactive visualization platform in Jupyter notebooks, epidemiologists, optimization practitioners and others (e.g. economists) can easily compare epidemiological models, costs functions and optimization algorithms to address important choicesto be made by health decision-makers. Trained models can be explored by experts and non-experts via a web interface.
 This article is part of the special track on AI and COVID-19.

Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: A Short Survey

“…The agent could explore in the space of constraints, setting constraints to itself, building a curriculum on these, etc. This is partially investigated in Colas et al (2021), where the agent samples constraint-based goals in the optimization of control strategies to mitigate the economic and health costs in simulated epidemics. This approach, however, only considers constraints on minimal values for the objectives and requires the training of an additional Q-function per constraint.…”

Modeling approaches for early warning and monitoring of pandemic situations as well as decision support

“…• those that retrospectively evaluate the effects of NPIs (26, 27, 31-34, 40, 48, 49, 56, 60, 97, 98), • those that make forecasts on the effects of a specified NPI in the sense of scenario planning (26, 31, 35, 38, 50-55, 58, 96), • and those that develop methods for optimal control policy identification (59,(100)(101)(102)(103)(104).…”

“…To find optimal control policies, offline RL strategies have been proposed by several authors. While Kwak et al ( 100 ) solely relied on deep learning and only focused on health aspects, other studies ( 101 – 104 ) focused on a hybrid modeling strategy incorporating an extended SEIR compartmental model for predicting potential NPI effects. Moreover, the latter studies incorporated the economic costs of NPIs as well.…”

Reinforcement Learning in Practice: Opportunities and Challenges

“…Capobianco et al (2021) study how to optimize mitigation policies considering both economic impact and hospital capacity. Colas et al (2021) propose a toolbox for optimizing control policies in epidemiological models. Trott et al (2021) propose to optimize economic and public policy, in particular, for COVID-19, with AI Economist (Zheng et al, 2021).…”