Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments

Abstract

Champion versus challenger field experiments are widely used to compare the performance of different targeting policies. These experiments randomly assign customers to receive marketing actions recommended by either the existing (champion) policy or the new (challenger) policy, and then compare the aggregate outcomes. We recommend an alternative experimental design and propose an alternative estimation approach to improve the evaluation of targeting policies. The recommended experimental design randomly assigns customers to marketing actions. This allows evaluation of any targeting policy without requiring an additional experiment, including policies designed after the experiment is implemented. The proposed estimation approach identifies customers for whom different policies recommend the same action and recognizes that for these customers there is no difference in performance. This allows for a more precise comparison of the policies. We illustrate the advantages of the experimental design and estimation approach using data from an actual field experiment. We also demonstrate that the grouping of customers, which is the foundation of our estimation approach, can help to improve the training of new targeting policies. This paper was accepted by Matthew Shum, marketing.

Design and Evaluation of Personalized Free Trials

“…A series of recent papers at the intersection of machine learning and causal inference have been developing methods to address these challenges and obtain individual-level treatment effects, which can then be used to personalize treatment assignment (Athey and Imbens, 2016;Wager and Athey, 2018). Similarly, a series of papers in marketing have combined powerful predictive machine learning models with experimental (or quasi-experimental) data to develop personalized targeting policies (Rafieian and Yoganarasimhan, 2020;Rafieian, 2019a,b;Simester et al, 2019a). At a high level, all these papers share the common goal of personalizing marketing interventions in order to maximize some measure of reward that is important to the firm.…”

Dynamic Marketing Policies: Constructing Markov States for Reinforcement Learning

“…The previous work has focused on validating myopic targeting problems. Simester et al, (2019b) propose the use of a randomizedby-action (RBA) design to improve the efficiency of model comparisons. Hitsch and Misra (2018) .…”

“…Finally, we may encounter covariate shift. Covariate shift arises if the distribution of the data used for training a targeting model is different than the data used for implementation (see for example Simester et al 2019b). In our setting, the probability vector at the last period, which will guide the targeting policy in a sequential targeting problem, might not occur in any prior period.…”

Parallel Experimentation on Advertising Platforms

“…While the focus of that literature is primarily on developing a scalable architecture, our focus is on developing a set of treatment effects that incorporate experimentation, and on ways in which treatment effects learned in the experiment can be extrapolated to other situations. This paper is related to the copious literature on measuring digital advertising effects via randomized controlled trials (e.g., Goldfarb and Tucker, 2011;Lewis and Reiley, 2014;Blake et al, 2015;Sahni, 2015;Sahni and Nair, 2016;Gordon et al, 2019), and to the empirical literature on measuring advertising effects in competition (e.g., Shapiro, 2018;Simester et al, 2019), though these papers have not addressed the issue of parallel experimentation to our knowledge. In addition, to the extent that we leverage counterfactual policy logging to improve the precision of our estimates, our work is related to the recent literature on digital advertising that has suggested such strategies for improving statistical efficiency (e.g., Johnson et al, 2017;Simester et al, 2019).…”

“…This ensures we implement statistical analysis on a set of users in the treatment group who have the highest opportunity to be factually served the focal ad, and on a set of equivalent users in the control group who have the highest opportunity to be counterfactually served the focal ad. By removing users with low propensity to be served the focal ad from the analysis, we improve precision (e.g., Johnson et al, 2017;Simester et al, 2019).…”