Improving the sensitivity of online controlled experiments by utilizing pre-experiment data

Deng, Alex; Xu, Ya; Kohavi, Ron; Walker, Toby

doi:10.1145/2433396.2433413

Cited by 173 publications

(116 citation statements)

References 11 publications

Supporting

Mentioning

115

Contrasting

Order By: Relevance

“…Better sensitivity is crucial in scaling an experiment system, as it effectively increases the number of experiments that can run concurrently without requiring more users. In Bing, we started using pre-experiment data to reduce the variance and improve sensitivity [45], but we believe there is room for significant improvements in this area.…”

Section: Resultsmentioning

confidence: 99%

Online controlled experiments at large scale

Kohavi

Deng

Frasca

et al. 2013

Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Self Cite

365

237

View full text Add to dashboard Cite

Web-facing companies, including Amazon, eBay, Etsy, Facebook, Google, Groupon, Intuit, LinkedIn, Microsoft, Netflix, Shop Direct, StumbleUpon, Yahoo, and Zynga use online controlled experiments to guide product development and accelerate innovation. At Microsoft's Bing, the use of controlled experiments has grown exponentially over time, with over 200 concurrent experiments now running on any given day. Running experiments at large scale requires addressing multiple challenges in three areas: cultural/organizational, engineering, and trustworthiness. On the cultural and organizational front, the larger organization needs to learn the reasons for running controlled experiments and the tradeoffs between controlled experiments and other methods of evaluating ideas. We discuss why negative experiments, which degrade the user experience short term, should be run, given the learning value and long-term benefits. On the engineering side, we architected a highly scalable system, able to handle data at massive scale: hundreds of concurrent experiments, each containing millions of users. Classical testing and debugging techniques no longer apply when there are billions of live variants of the site, so alerts are used to identify issues rather than relying on heavy upfront testing. On the trustworthiness front, we have a high occurrence of false positives that we address, and we alert experimenters to statistical interactions between experiments. The Bing Experimentation System is credited with having accelerated innovation and increased annual revenues by hundreds of millions of dollars, by allowing us to find and focus on key ideas evaluated through thousands of controlled experiments. A 1% improvement to revenue equals more than $10M annually in the US, yet many ideas impact key metrics by 1% and are not well estimated a-priori. The system has also identified many negative features that we avoided deploying, despite key stakeholders' early excitement, saving us similar large amounts.

show abstract

Section: Resultsmentioning

confidence: 99%

Online controlled experiments at large scale

Kohavi

Deng

Frasca

et al. 2013

Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Self Cite

365

237

View full text Add to dashboard Cite

show abstract

“…More sophisticated variation reduction techniques were proposed in [7]: the stratification and the usage of control variates based on pre-experiment data. The authors of [16] noted that the reduction of skewness of the evaluation metric used in an A/B experiment could also improve the sensitivity (e.g., transformation of the metric or capping its values [16]).…”

Section: Related Workmentioning

confidence: 99%

“…Both approaches have disadvantages. First, the amount of data is upper-bounded by the available web service traffic, and the growth of traffic per experiment reduces the number of experiments conducted per year [28,7]. Second, increasing the size of a treatment group in the treatment group is always not desirable, as any treatment may have a negative effect 2 .…”

Section: Introductionmentioning

confidence: 99%

“…The variance reduction techniques (the stratification and control variates), used in Monte Carlo simulations, were also applied [7] in the A/B testing. Though, they do not have the above-mentioned drawbacks and lead to a noticeable variance reduction, these techniques are based on the utilization of pre-experiment data and, hence, are limited in their applicability as well.…”

Section: Introductionmentioning

confidence: 99%

“…To the best of our knowledge, no existing study on user engagement investigated such a prediction problem 4 . Finally, our approach could be used simultaneously with other sensitivity improvement methods (i.e., with stratification, control variates [7], transformation of the metric, filtration of the treatment group [18], etc.) and, hence, is not an a substitute, but rather a complement to them.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Drutsa

Gusev

Serdyukov

2015

Proceedings of the 24th International Conference on World Wide Web

View full text Add to dashboard Cite

Modern Internet companies improve their services by means of data-driven decisions that are based on online controlled experiments (also known as A/B tests). To run more online controlled experiments and to get statistically significant results faster are the emerging needs for these companies. The main way to achieve these goals is to improve the sensitivity of A/B experiments. We propose a novel approach to improve the sensitivity of user engagement metrics (that are widely used in A/B tests) by utilizing prediction of the future behavior of an individual user. This problem of prediction of the exact value of a user engagement metric is also novel and is studied in our work. We demonstrate the effectiveness of our sensitivity improvement approach on several real online experiments run at Yandex. Especially, we show how it can be used to detect the treatment effect of an A/B test faster with the same level of statistical significance.

show abstract

Is designed data collection still relevant in the big data era? – A discussion

Stevens

2023

Quality & Reliability Eng

View full text Add to dashboard Cite

design and analysis of experiments, hyperparameter tuning, simulation design I would first like to congratulate Dr. Anderson-Cook and Dr. Lu for an important, insightful, and timely paper. I'm honored and grateful to have been invited to discuss the paper and share my thoughts on this topic. Although I must admit, I find it difficult to write a discussion because I agree very strongly with their arguments. In particular, I too emphatically believe that designed data collection remains relevant in the era of Big Data. In what follows I offer my thoughts on opportunities for designed experiments in this modern era, amplifying and augmenting some mentioned in the main paper, and also remarking on others not addressed.Drs. Anderson-Cook and Lu advocate for the use of design of experiment (DOE) principles in the execution and analysis of simulation studies. Chipman and Bingham 1 have recently provided similar guidance, acknowledging that the simulation studies now ubiquitous in methodological statistics and data science research tend to be factorial experiments, whether the researchers identify them as such or not. Chipman and Bingham 1 describe the value of using fractional factorial designs to more efficiently study the simulation factors, and they demonstrate how ANOVA and main and interaction effect plots may be used to provide deeper insights into the results of a simulation. In the interest of developing widely useful methodology, they also encourage the use of robust parameter designs for simulations to study the robustness of methodological developments to uncontrollable factors.In a similar vein, another contemporary application of experimental design principles is in machine learning hyperparameter optimization. The models and algorithms associated with machine learning and deep learning efforts typically rely on several (in some cases dozens of) hyperparameters and interest lies in identifying the configuration of those parameters that optimizes predictive performance. Most often, hyperparameter tuning is done via grid search (i.e., factorial designs) but this problem is perfectly suited for response surface methodology to both efficiently determine which hyperparameters are important (via screening experiments) and then to determine the optimal configuration of the important ones via response surface experiments and second order models. Lujan-Moreno et al., 2 Zhang et al., 3 and Pannakkong et al. 4 illustrate these ideas in the contexts of random forests, support vector machines, and neural networks, but this is still a burgeoning application area; there is no clear consensus on how hyperparameter values should be tuned, so more work and wider advocacy of DOE methods in this area is warranted. 5 Another DOE application of interest to data scientists that Drs. Anderson-Cook and Lu identified is online controlled experimentation (OCE). Such experiments, colloquially referred to as A/B tests, are typically performed to improve and innovate internet-based products and services, ultimately aiming to maximize ...

show abstract

Improving the sensitivity of online controlled experiments by utilizing pre-experiment data

Cited by 173 publications

References 11 publications

Online controlled experiments at large scale

Online controlled experiments at large scale

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Is designed data collection still relevant in the big data era? – A discussion

Contact Info

Product

Resources

About