Field Study in Deploying Restless Multi-Armed Bandits: Assisting Non-profits in Improving Maternal and Child Health

Mate, Aditya; Madaan, Lovish; Taneja, Aparna; Madhiwalla, Neha; Verma, Shresth; Singh, Gargi; Hegde, Aparna; Varakantham, Pradeep; Tambe, Milind

doi:10.1609/aaai.v36i11.21460

Cited by 18 publications

(39 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We briefly note here that unless the MDP's state space C is relatively small, then the (data-dependent) permutation set Π 4 will typically be quite small and virtually no randomization in permutations can be performed. There are, however, settings in which the state space C is small, such as in the restless multiarmed bandit considered by Mate et al (2020Mate et al ( , 2021 in the context of patient adherence monitoring and well-being, in which our test can be used effectively.…”

Section: Conditional Independence Testingmentioning

confidence: 99%

Randomization Tests for Adaptively Collected Data

Nair¹,

Janson²

2023

Preprint

View full text Add to dashboard Cite

Randomization tests (including permutation tests) are one of the most fundamental methods in statistics, enabling a range of inferential tasks such as testing for (conditional) independence of random variables, constructing confidence intervals in semiparametric location models, and constructing (by inverting a permutation test) model-free prediction intervals via conformal inference. Randomization tests are intuitive, easy to implement, and exactly valid for any sample size, but their use is generally confined to independent and/or exchangeable data. Yet in many applications including clinical trials, online education, online advertising, and protein design, data is routinely collected adaptively, meaning that the aspects of the data under the data collector's control (e.g., treatment assignments) are assigned at each time step via a (possibly randomized) algorithm that depends on all the data observed so far; such assignment algorithms include (contextual) bandit and reinforcement learning algorithms as well as adaptive experimental designs. In this paper we present a general framework for randomization testing on adaptively collected data (despite its non-exchangeability), encompassing (and in some cases improving) the few existing results on randomization testing and conformal inference for adaptively collected data, as well as many other important settings. The key to our framework is the ability to compute likelihoodratio-based weights involving known quantities based purely on the known adaptive assignment algorithm, as long as a certain proportionality condition is met. These weights can then be accounted for in our framework to conduct an exact randomization test, but in order for the test to be powerful, resamples need to be diverse yet have weights as close to equal as possible. Thus, we additionally present novel computationally tractable resampling algorithms for various popular adaptive assignment algorithms, data-generating environments, and types of inferential tasks. Finally, we demonstrate via a range of simulations our framework's power (in the case of hypothesis testing) and narrow widths (in the case of confidence or prediction intervals produced by inverting randomization tests).

show abstract

Section: Conditional Independence Testingmentioning

confidence: 99%

Randomization Tests for Adaptively Collected Data

Nair¹,

Janson²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Our previous study conducted in April 2021 Mate et al. (2022) is the first to present real‐world service quality improvement using RMABs in the context of mMitra program. This study tested an RMAB‐based policy against two baselines of interest, and showed RMAB outperforming its competitors.…”

Section: Restless Multi‐armed Bandits (Rmab)mentioning

confidence: 99%

“…Furthermore, whereas RMAB achieves statistically significant improvement against CSOC (

p &lt; 0.05

) and RR (

p &lt; 0.1

), RR fails to achieve any statistically significant improvement over CSOC (see Mate et al. 2022 for more details). This key result forms the basis of relying on RMAB‐based strategy over other non‐AI strategies as a basis of Saheli .…”

Section: Restless Multi‐armed Bandits (Rmab)mentioning

confidence: 99%

See 1 more Smart Citation

Expanding impact of mobile health programs: SAHELI for maternal and child care

Verma,

Singh,

Mate

et al. 2023

AI Magazine

Self Cite

View full text Add to dashboard Cite

Underserved communities face critical health challenges due to lack of access to timely and reliable information. Nongovernmental organizations are leveraging the widespread use of cellphones to combat these healthcare challenges and spread preventative awareness. The health workers at these organizations reach out individually to beneficiaries; however, such programs still suffer from declining engagement. We have deployed Saheli, a system to efficiently utilize the limited availability of health workers for improving maternal and child health in India. Saheli uses the Restless Multi‐armed Bandit (RMAB) framework to identify beneficiaries for outreach. It is the first deployed application for RMABs in public health, and is already in continuous use by our partner NGO, ARMMAN. We have already reached ∼130K beneficiaries with Saheli, and are on track to serve one million beneficiaries by the end of 2023. This scale and impact has been achieved through multiple innovations in the RMAB model and its development, in preparation of real world data, and in deployment practices; and through careful consideration of responsible AI practices. Specifically, in this paper, we describe our approach to learn from past data to improve the performance of Saheli's RMAB model, the real‐world challenges faced during deployment and adoption of Saheli, and the end‐to‐end pipeline.

show abstract

“…Bandit optimization is a very general framework for sequential decision making when the dynamics of the underlying environment are unknown a priori. It has been well studied over the past few decades, and has shown great empirical success in areas including ad placement, clinical trials [3,4]. Such standard bandit methods often assume the decision maker has access to the entire user context or user state.…”

Section: Introductionmentioning

confidence: 99%

Optimal Algorithms for Latent Bandits with Cluster Structure

Pal¹,

Suggala²,

Shanmugam³

et al. 2023

Preprint

View full text Add to dashboard Cite

We consider the problem of latent bandits with cluster structure where there are multiple users, each with an associated multi-armed bandit problem. These users are grouped into latent clusters such that the mean reward vectors of users within the same cluster are identical. At each round, a user, selected uniformly at random, pulls an arm and observes a corresponding noisy reward. The goal of the users is to maximize their cumulative rewards. This problem is central to practical recommendation systems and has received wide attention of late [1,2]. Now, if each user acts independently, then they would have to explore each arm independently and a regret of Ω( √ MNT) is unavoidable, where M, N are the number of arms and users, respectively. Instead, we propose LATTICE (Latent bAndiTs via maTrIx ComplEtion) which allows exploitation of the latent cluster structure to provide the minimax optimal regret of O( (M + N)T), when the number of clusters is Õ(1). This is the first algorithm to guarantee such strong regret bound. LATTICE is based on a careful exploitation of arm information within a cluster while simultaneously clustering users. Furthermore, it is computationally efficient and requires only O(log T) calls to an offline matrix completion oracle across all T rounds.

show abstract

Field Study in Deploying Restless Multi-Armed Bandits: Assisting Non-profits in Improving Maternal and Child Health

Cited by 18 publications

References 33 publications

Randomization Tests for Adaptively Collected Data

Randomization Tests for Adaptively Collected Data

Expanding impact of mobile health programs: SAHELI for maternal and child care

Optimal Algorithms for Latent Bandits with Cluster Structure

Contact Info

Product

Resources

About