Markovian And Non-Markovian Processes with Active Decision Making Strategies For Addressing The COVID-19 Pandemic

Abstract

We study and predict the evolution of Covid-19 in six US states from the period May 1 through August 31 using a discrete compartment-based model and prescribe active intervention policies, like lockdowns, on the basis of minimizing a loss function, within the broad framework of partially observed Markov decision processes. For each state, Covid-19 data for 40 days (starting from May 1 for two northern states and June 1 for four southern states) are analyzed to estimate the transition probabilities between compartments and other parameters associated with the evolution of the epidemic.These quantities are then used to predict the course of the epidemic in the given state for the next 50 days (test period) under various policy allocations, leading to different values of the loss function over the training horizon. The optimal policy allocation is the one corresponding to the smallest loss. Our analysis shows that none of the six states need lockdowns over the test period, though the 'no lockdown' prescription is to be interpreted with caution: responsible mask use and social distancing of course need to be continued. The caveats involved in modeling epidemic propagation of this sort are discussed at length. A sketch of a non-Markovian formulation of Covid-19 propagation (and more general epidemic propagation) is presented as an attractive avenue for future research in this area.

Multi-Objective Model-based Reinforcement Learning for Infectious Disease Control

“…Therefore, the model-based approach is adopted in this paper. Finally, we note that some efforts have been made in the literature to study the optimal control policy for COVID-19 (Alvarez, Argente, and Lippi 2020;Piguillem and Shi 2020;Eftekhari et al 2020). As discussed above, these works did not consider the multi-objective problem but assigned a fixed price for each life to scalarize the objective, and in addition, they focus on solving a one-time planning problem to decide all future actions instead of providing a real-time decision-making framework.…”

Reinforcement Learning in Possibly Nonstationary Environments

“…As another example, the coronavirus disease 2019 (COVID-19) has been one of the worst global pandemics in history affecting millions of people. There is a growing interest in ap-plying RL to develop data-driven intervention policies to contain the spread of the virus (see e.g., Eftekhari et al, 2020;Kompella et al, 2020;Wan et al, 2020). However, the spread of COVID-19 is an extremely complex process and is nonstationary over time.…”