Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework

Abstract

A/B testing, or online experiment is a standard business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries. Major challenges arise in online experiments of two-sided marketplace platforms (e.g., Uber) where there is only one unit that receives a sequence of treatments over time. In those experiments, the treatment at a given time impacts current outcome as well as future outcomes. The aim of this article is to introduce a reinforcement learning framework for carrying A/B testing in these experiments, while characterizing the long-term treatment effects. Our proposed testing procedure allows for sequential monitoring and online updating. It is generally applicable to a variety of treatment designs in different industries. In addition, we systematically investigate the theoretical properties (e.g., size and power) of our testing procedure. Finally, we apply our framework to both simulated data and a real-world data example obtained from a technological company to illustrate its advantage over the current practice. A Python implementation of our test is available at https://github.com/callmespring/ CausalRL.

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes

“…OPE via causal inference. Our work is closely related to the line of research that employing tools from causal inference (Pearl, 2009) for studying OPE with unobserved confounders (Oberst and Sontag, 2019;Kallus and Zhou, 2020;Kallus and Zhou, 2021;Mastouri et al, 2021;Shi et al, 2021;Shi et al, 2022). Among them, ; Shi et al (2021) are most relevant to our work.…”

Markovian Interference in Experiments

“…We study a generic experimentation problem within a system represented as a Markov Decision Process (MDP), where treatment corresponds to an action which may interfere with state transitions. This form of interference, which we refer to as Markovian, naturally subsumes the platform examples above, as recently noted by others either implicitly [50] or explicitly [29,55]. In that example, a user arrives at each time step, the platform chooses an action (whether to treat the user), and the user's purchase decision alters the system state (inventory levels).…”

“…This tack appears to be promising, e.g. [55], but we observe that the resulting variance is necessarily large (Theorem 3).…”

“…This natural approach appears to be promising, e.g. [55], but outside of secondary issues (e.g. discounted vs. average reward), the primary issue is that being unbiased appears to come at a price: variance.…”

Rate-Optimal Contextual Online Matching Bandit

“…Bandit algorithms (Bubeck and Cesa-Bianchi, 2012;Lattimore and Szepesvári, 2020) and reinforcement learning (Sutton and Barto, 2018) are modern strategies to solve sequential decision making problems. They have received recent attentions in statistics community for business and scientific applications including dynamic pricing (Wang et al, 2020;Chen, Simchi-Levi and Wang, 2021;Chen, Miao and Wang, 2021;Wang et al, 2021), online decision making (Shi et al, 2020;Chen, Lu and Song, 2021;Chen et al, 2022), dynamic treatment regimes (Qi and Liu, 2018;Luckett et al, 2019;Qi et al, 2020;Qi, Miao and Zhang, 2021), and online causal effect in two-sided market (Shi et al, 2022b).…”