A simulation-deep reinforcement learning (SiRL) approach for epidemic control optimization

Abstract

In this paper, we address the controversies of epidemic control planning by developing a novel Simulation-Deep Reinforcement Learning (SiRL) model. COVID-19 reminded constituents over the world that government decision-making could change their lives. During the COVID-19 pandemic, governments were concerned with reducing fatalities as the virus spread but at the same time also maintaining a flowing economy. In this paper, we address epidemic decision-making regarding the interventions necessary given of the epidemic based on the purpose of the decision-maker. Further, we intend to compare different vaccination strategies, such as age-based and random vaccination, to shine a light on who should get priority in the vaccination process. To address these issues, we propose a simulation-deep reinforcement learning (DRL) framework. This framework is composed of an agent-based simulation model and a governor DRL agent that can enforce interventions in the agent-based simulation environment. Computational results show that our DRL agent can learn effective strategies and suggest optimal actions given a specific epidemic situation based on a multi-objective reward structure. We compare our DRL agent’s decisions to government interventions at different periods of time during the COVID-19 pandemic. Our results suggest that more could have been done to control the epidemic. In addition, if a random vaccination strategy that allows super-spreaders to get vaccinated early were used, infections would have been reduced by 32% at the expense of 4% more deaths. We also show that a behavioral change of fully quarantining 10% of the risky individuals and using a random vaccination strategy leads to a reduction of the death toll by 14% and 27% compared to the age-based vaccination strategy that was implemented and the New Jersey reported data, respectively. We have also demonstrated the flexibility of our approach to be applied to other locations by validating and applying our model to the COVID-19 case in the state of Kansas.

Learning from the COVID-19 pandemic: a systematic review of mathematical vaccine prioritization models

“…This study concludes that the answer primarily depends on the speed of the vaccine roll-out, as well as the differences in NPI compliance. Several other studies come to the same or similar conclusion (Bushaj et al, 2023). An ABM-based study employs a binary stratification of NPI compliance, and quantifies how much lower the compliance level of low-risk (versus high-risk) individuals must be for them to be the optimal prioritization target (Kadelka and McCombs, 2021).…”

“…All investigated studies chose a ∈ (0, 1]. The transmission rate β i can account for age-dependent susceptibility and risk mitigation (e.g., mask wearing). Multiple studies assumed that older, more vulnerable individuals suffer from higher susceptibility (Davies et al, 2020; Moore et al, 2021b; Jahn et al, 2021) but also engage in more risk mitigation measures (Masters et al, 2020; Kadelka and McCombs, 2021; Bushaj et al, 2023; Vo et al, 2023). The transmission rate may also vary over time, e.g., due to the emergence of more transmissible SARS-CoV-2 variants (Islam et al, 2021; Moore et al, 2021b), or time-varying social distancing levels (Moore et al, 2021b).…”

“…In Bushaj et al (2023), the Covasim ABM from Kerr et al (2021) was expanded to compare a random with an age-structured vaccine allocation strategy. The authors show that a “governor Deep Reinforcement Learning agent” can learn effective strategies and suggest, based on a multi-objective reward structure, optimal ABM interventions when presented with a specific epidemic situation.…”

A K-means Supported Reinforcement Learning Framework to Multi-dimensional Knapsack

“…Some more recent studies use DRL to learn from state-of-the-art algorithms and improve them further [21,45,65,80,85]. Bushaj et al [11] present a simulation deep reinforcement learning framework that avoids using COP for epidemic control optimization, instead using a reinforcement learning algorithm as a decision maker.…”

How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making

“…The point of difference with other machine learning methods is that in reinforcement learning, no data need to be given in advance; the learning information is obtained by receiving rewards or feedback from the environment for the actions, which enables the updating of the model parameters. As a class of framework methods, it has a certain degree of versatility; it can be widely integrated in such areas as cybernetics [4], game theory [5], information theory [6], operations research [7], simulation optimization [8], multiagent systems [9], collective intelligence [10], statistics [11], and other fields of results; and it is very suitable for complex intelligent decision-making problems. Compared to centralized intelligent decision making, the approach of syncing the computation and decision-making processes in reinforcement learning at the edge has additional advantages.…”