Online controlled experiments at large scale

Kohavi, Ron; Deng, Alex; Frasca, Brian; Walker, Toby; Xu, Ya; Pohlmann, Nils

doi:10.1145/2487575.2488217

Cited by 360 publications

(240 citation statements)

References 15 publications

Supporting

Mentioning

237

Contrasting

Unclassified

Order By: Relevance

“…The theoretical details of the online controlled experiment methodology were widely described in the existing works [22,17,18], and we conduct our experiments in accordance with them. A rich practical experience of using this methodology for different evaluation cases in many web companies was studied recently [13,19,28,15]. These studies concern, inter alia, experiments with different components of a web service (the user interface [13,8], ranking algorithms [27,8], etc.…”

Section: Related Workmentioning

confidence: 99%

“…It is utilized by many web companies, e.g., web search engines such as Bing [6,16] and Google [28], social networks like Facebook [1], etc. The largest ones have designed special experimental platforms that allow them to run A/B tests at large scale (e.g., hundreds of concurrent experiments per day) [28,15].…”

Section: Introductionmentioning

confidence: 99%

“…The applicability of these approaches is limited and strongly depends on the case of a conducted experiment. Namely, an experimental platform usually has a standardized success criteria, which utilizes a small set of key metrics to make the final decision on the treatment variant of the service [15]. These metrics are usually selected with respect to some business-related criteria of a considered service and are aligned with its long-term goals (like the number of sessions per user for a search engine [14]).…”

Section: Introductionmentioning

confidence: 99%

“…User engagement reflects how often the user solves her needs (e.g., to search something) by means of the considered service (e.g., a search engine). On the one hand, these metrics are measurable in the short-term experiment period, and, on the other hand, they are predictive of the long-term success of the company [14,15,16,25]. That is why engagement metrics are often considered to be most appropriate for online evaluation.…”

Section: Introductionmentioning

confidence: 99%

“…That is why engagement metrics are often considered to be most appropriate for online evaluation. In this study, we pay special attention to the metrics that reflect the loyalty aspect of engagement: the state-of-the-art number of sessions per user metric [14,27] (which is accepted as the "North-star" for A/B testing evaluation in major search engine companies like Bing [15,16]) and the recently proposed absence time metric [9]. Our approach raises the problem of prediction of the future individual user engagement.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Drutsa

Gusev

Serdyukov

2015

Proceedings of the 24th International Conference on World Wide Web

View full text Add to dashboard Cite

Modern Internet companies improve their services by means of data-driven decisions that are based on online controlled experiments (also known as A/B tests). To run more online controlled experiments and to get statistically significant results faster are the emerging needs for these companies. The main way to achieve these goals is to improve the sensitivity of A/B experiments. We propose a novel approach to improve the sensitivity of user engagement metrics (that are widely used in A/B tests) by utilizing prediction of the future behavior of an individual user. This problem of prediction of the exact value of a user engagement metric is also novel and is studied in our work. We demonstrate the effectiveness of our sensitivity improvement approach on several real online experiments run at Yandex. Especially, we show how it can be used to detect the treatment effect of an A/B test faster with the same level of statistical significance.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Drutsa

Gusev

Serdyukov

2015

Proceedings of the 24th International Conference on World Wide Web

View full text Add to dashboard Cite

show abstract

Learning causality with graphs

2022

AI Magazine

View full text Add to dashboard Cite

Recent years have witnessed a rocketing growth of machine learning methods on graph data, especially those powered by effective neural networks. Despite their success in different real‐world scenarios, the majority of these methods on graphs only focus on predictive or descriptive tasks, but lack consideration of causality. Causal inference can reveal the causality inside data, promote human understanding of the learning process and model prediction, and serve as a significant component of artificial intelligence (AI). An important problem in causal inference is causal effect estimation, which aims to estimate the causal effects of a certain treatment (e.g., prescription of medicine) on an outcome (e.g., cure of disease) at an individual level (e.g., each patient) or a population level (e.g., a group of patients). In this paper, we introduce the background of causal effect estimation from observational data, envision the challenges of causal effect estimation with graphs, and then summarize representative approaches of causal effect estimation with graphs in recent years. Furthermore, we provide some insights for future research directions in related area. Link to video abstract: https://youtu.be/BpDPOOqw-ns

show abstract

A fast bootstrap algorithm for causal inference with large data

Kosko,

Wang,

Santacatterina

2024

Statistics in Medicine

View full text Add to dashboard Cite

Estimating causal effects from large experimental and observational data has become increasingly prevalent in both industry and research. The bootstrap is an intuitive and powerful technique used to construct standard errors and confidence intervals of estimators. Its application however can be prohibitively demanding in settings involving large data. In addition, modern causal inference estimators based on machine learning and optimization techniques exacerbate the computational burden of the bootstrap. The bag of little bootstraps has been proposed in non‐causal settings for large data but has not yet been applied to evaluate the properties of estimators of causal effects. In this article, we introduce a new bootstrap algorithm called causal bag of little bootstraps for causal inference with large data. The new algorithm significantly improves the computational efficiency of the traditional bootstrap while providing consistent estimates and desirable confidence interval coverage. We describe its properties, provide practical considerations, and evaluate the performance of the proposed algorithm in terms of bias, coverage of the true 95% confidence intervals, and computational time in a simulation study. We apply it in the evaluation of the effect of hormone therapy on the average time to coronary heart disease using a large observational data set from the Women's Health Initiative.

show abstract

Online controlled experiments at large scale

Cited by 360 publications

References 15 publications

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Learning causality with graphs

A fast bootstrap algorithm for causal inference with large data

Contact Info

Product

Resources

About