Design and Evaluation of Personalized Free Trials

Yoganarasimhan, Hema; Barzegary, Ebrahim; Pani, Abhishek

doi:10.48550/arxiv.2006.13420

Cited by 1 publication

(4 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The total 3-year revenue impact, relative to the status quo, of treating the first two cohorts with the policy optimized using the surrogate index sums to $4-5 million. Our paper adds to and complements a recent and growing literature in marketing on policy evaluation and learning (e.g., Hitsch and Misra, 2018;Simester et al, 2019a,b;Yoganarasimhan et al, 2020) and empirical work in proactive churn management (e.g., Ascarza, 2018) by focusing on optimizing targeting policies for long-term retention and revenue.…”

Section: Discussionmentioning

confidence: 76%

“…6 Second, we systematically add randomized exploration around the learned policy, which allows us to evaluate and update the policy for future cohorts in case the environment changes. Hitsch and Misra (2018) and Yoganarasimhan et al (2020) studied the problem in a static setting. Simester et al (2019b) did look at changes in the environment but they focused on evaluating the robustness of different machine learning models.…”

Section: Related Workmentioning

confidence: 99%

“…Simester et al (2019b) did look at changes in the environment but they focused on evaluating the robustness of different machine learning models. Third, we use a doubly-robust (DR) approach (Dudík et al, 2014) for both policy evaluation and learning in contrast to Hitsch and Misra (2018) and Yoganarasimhan et al (2020) who used an inverse probability weighting (IPW) estimator for policy evaluation. Lemmens and Gupta (2020) introduce a specialized incremental-profit-based loss function that performs well in their empirical evaluation, but lacks the asymptotic efficiency results available for doubly-robust policy learning; it is also unclear how to combine this with known probabilities of treatment (i.e., design-based propensity scores) that arise in sophisticated experiments.…”

Section: Related Workmentioning

confidence: 99%

“…Yoganarasimhan et al (2020) showed in their particular case the policy learned on short-term outcome also does well on long-term outcomes, but the policy is not directly optimized on long-term outcome.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Targeting for long-term outcomes

Yang¹,

Eckles²,

Dhillon³

et al. 2020

Preprint

View full text Add to dashboard Cite

Decision-makers often want to target interventions (e.g., marketing campaigns) so as to maximize an outcome that is observed only in the long-term. This typically requires delaying decisions until the outcome is observed or relying on simple short-term proxies for the longterm outcome. Here we build on the statistical surrogacy and off-policy learning literature to impute the missing long-term outcomes and then approximate the optimal targeting policy on the imputed outcomes via a doubly-robust approach. We apply our approach in large-scale proactive churn management experiments at The Boston Globe by targeting optimal discounts to its digital subscribers to maximize their long-term revenue. We first show that conditions for validity of average treatment effect estimation with imputed outcomes are also sufficient for valid policy evaluation and optimization; furthermore, these conditions can be somewhat relaxed for policy optimization. We then validate this approach empirically by comparing it with a policy learned on the ground truth long-term outcomes and show that they are statistically indistinguishable. Our approach also outperforms a policy learned on short-term proxies for the long-term outcome. In a second field experiment, we implement the optimal targeting policy with additional randomized exploration, which allows us to update the optimal policy for each new cohort of customers to account for potential non-stationarity. Over three years, our approach had a net-positive revenue impact in the range of $4-5 million compared to The Boston Globe's current policies.

show abstract

Section: Discussionmentioning

confidence: 76%

Section: Related Workmentioning

confidence: 99%