When Less Is More: Data and Power in Advertising Experiments

Abstract

Yahoo! Research partnered with a nationwide retailer to study the effects of online display advertising on both online and in-store purchases. We use a randomized field experiment on 3 million Yahoo! users who are also past customers of the retailer. We find statistically significant evidence that the retailer ads increase sales 3.6% relative to the control group. We show that control ads boost measurement precision by identifying and removing the half of in-campaign sales data that is unaffected by the ads. Less data gives us 31% more precision in our estimates-equivalent to increasing our sample to 5.3 million users. By contrast, we only improve precision by 5% when we include additional covariate data to reduce the residual variance in our experimental regression. The covariate-adjustment strategy disappoints despite exceptional consumer-level data including demographics, ad exposure levels, and two years' worth of past purchase history.

Vitamin panacea: Is advertising fueling demand for products with uncertain scientific benefit?

“…If a consumer moved from seeing zero to the mean level of print advertising (just 3 additional ads), we would expect a 1.2 percentage point increase in the probability of consumption, or a 2 percent increase off a base consumption rate of 58.9 percent. The results are in line with what has been found in studies examining other DTCA markets (antidepressants 3 percentage points found by Avery et al, 2012; smokeless tobacco 2.5 percentage points found by Dave and Saffer 2013) and advertising experiments (Johnson et al 2016). Given the monetary cost of the good and the possible health consequences (Swartzberg 2016), these small effects might still have a large impact on public health.…”

“…Since some product purchasing is rare and the effect of advertising likely small, very large sample sizes may be needed in order to have sufficient power to detect effects. Johnson et al (2016) studied one of these field experiments with 3 million Yahoo! users by randomly exposing a treatment group to retailer specific advertisements.…”

Advertising in Health Insurance Markets

“…Despite evidence of cream skimming and social mis-allocation, this paper shows evidence that further regulation of television advertising is unlikely to solve any of these problems. This paper also adds to a growing literature on advertising effectiveness, some of which documents similarly small advertising effects (e.g., Blake, Nosko and Tadelis, 2015), while others document significant effects, even in the presence of the aforementioned principle-agent issues in the industry (e.g., Johnson, Lewis and Reiley, 2015;Shapiro, 2016). On the measurement side, all of these studies have shown that failure to consider the endogeneity of advertising can lead to large biases in estimated advertising effects, usually in the upward direction.…”

“…Documentation of this problem along with attempts at at a solution date back at least to Lodish et al (1995), which uses split cable experiments to think about advertising effectiveness. Additionally, a recent stream of literature has used either field experiments online (e.g., Blake, Nosko and Tadelis, 2015;Johnson, Lewis and Reiley, 2015;Sahni, 2015a,b;Lewis and Nguyen, 2015), instrumental variables (e.g., Sinkinson and Starc, 2015;Hartmann and Klapper, 2014) or natural experiments (e.g., Shapiro, 2016;Tuchman, 2015;Spenkuch and Toniatti, 2015). This paper will follow the latter strategy and exploit the random nature of TV market borders.…”

Approximating Purchase Propensities and Reservation Prices From Broad Consumer Tracking

“…Subsequent proposed bills have not yet been enacted, as of August 2019.4 Robinson (1934) notes that price discrimination can increase aggregate surplus only if sales rise. If sales remain the same (or fall), then aggregate surplus falls because some goods are (inevitably) inefficiently allocated to consumers that have a lower willingness to pay than some unserved consumers offered higher prices.5 Others have noted that measuring online advertising effectiveness is inherently difficult(Lewis and Rao, 2015;Johnson et al, 2016).6 Hannak et al (2014) andMikians et al (2012) found some evidence of personalized pricing online but did not study its effectiveness Rossi et al (1995Rossi et al ( , 1996. andWaldfogel (2015) do investigate personalized prices but do not use detailed web-browsing histories as explanatory variables.7 Personalized marketing, including pricing, is often referred to in the marketing literature as "customer addressability.…”