Uplift Modeling with Multiple Treatments and General Response Types

Abstract

Randomized experiments have been used to assist decisionmaking in many areas. They help people select the optimal treatment for the test population with certain statistical guarantee. However, subjects can show significant heterogeneity in response to treatments. The problem of customizing treatment assignment based on subject characteristics is known as uplift modeling, differential response analysis, or personalized treatment learning in literature. A key feature for uplift modeling is that the data is unlabeled. It is impossible to know whether the chosen treatment is optimal for an individual subject because response under alternative treatments is unobserved. This presents a challenge to both the training and the evaluation of uplift models. In this paper we describe how to obtain an unbiased estimate of the key performance metric of an uplift model, the expected response. We present a new uplift algorithm which creates a forest of randomized trees. The trees are built with a splitting criterion designed to directly optimize their uplift performance based on the proposed evaluation method. Both the evaluation method and the algorithm apply to arbitrary number of treatments and general response types. Experimental results on synthetic data and industry-provided data show that our algorithm leads to significant performance improvement over other applicable methods.

A Practically Competitive and Provably Consistent Algorithm for Uplift Modeling

“…The performance of UCTS is tested under different training data sizes, specifically, 500, 2000, 4000, 8000, 16000, and 32000 samples per treatment. For each size, 10 training sets and test sets are provided in [15]. We use the results from these data to generate the 95% margin of error.…”

“…According to the results on the validation set (30% of training data), we set rho = 0.45 and min_split = 5. Details on parameter tuning of the 6 other methods can be found in the Appendix of [15].…”

“…Disappointed by the performance of the Separate Model Approach, researchers have proposed a number of specialized algorithms for uplift modeling. Most of them are designed for the special case of a single treatment [4] [14] and [15]. In [13], the tree-based algorithm described in [9] is extended to multiple treatment cases by using a weighted sum of pairwise distributional divergence as the splitting criterion.…”

Uplift Modeling for Multiple Treatments with Cost Optimization

“…In this section, the proposed models are evaluated empirically with both synthetic data and real data examples. This study also includes models that have been extended to the multiple treatment groups setting previously [4], [8].…”

“…Our evaluation metric is based on the Area Under the Uplift Curve (AUUC). [2], [4], [5], [8] This metric is calculated by sorting the observations in the testing set to 100 bins from the highest predicted uplift to the lowest. Because the treatment assignment is randomized, we have an approximately equal amount of treatment and control observation in each bin, which allows us to calculate the average treatment effect within each bin.…”