Top Challenges from the first Practical Online Controlled Experiments Summit

Abstract

Online controlled experiments (OCEs), also known as A/B tests, have become ubiquitous in evaluating the impact of changes made to software products and services. While the concept of online controlled experiments is simple, there are many practical challenges in running OCEs at scale. To understand the top practical challenges in running OCEs at scale and encourage further academic and industrial exploration, representatives with experience in large-scale experimentation from thirteen different organizations (Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University) were invited to the first Practical Online Controlled Experiments Summit. All thirteen organizations sent representatives. Together these organizations have tested more than one hundred thousand experiment treatments last year. Thirty-four experts from these organizations participated in the summit in Sunnyvale, CA, USA on December 13-14, 2018. While there are papers from individual organizations on some of the challenges and pitfalls in running OCEs at scale, this is the first paper to provide the top challenges faced across the industry for running OCEs at scale and some common solutions.

Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics

“…Under the same assumptions as required for the mediation case this would lead to testable restrictions, but the typical use case is a different one. As in [Athey, Chetty, Imbens, and Kang, 2016], a prominent use case is that with two samples (see also [Gupta, Kohavi, Tang, and Xu, 2019] In the surrogate case, as in the mediation case, the DAG can clarify the content of the assumptions. In particular it rules out a direct effect of the treatment on the outcome (as in Figure 7(a)).…”

Statistical Significance,p-Values, and the Reporting of Uncertainty

“…Consider a decision maker choosing whether to implement a new policyperhaps mandating a new early childhood educational program (Krueger and Whitmore 2001;Schanzenbach 2006;Chetty et al 2011), or making micro credit available to communities in developing countries (Banerjee, Karlan, and Zinman 2015;Crépon et al 2015;Meager 2019), or changing a search algorithm for a tech company (Gomez-Uribe and Hunt 2015; Gupta et al 2019). Suppose the only unknown component of the utility of implementing the policy is the average treatment effect (the difference in the average outcome if everybody was exposed to the intervention versus the average outcome if nobody was exposed).…”

“…There would also be a discussion regarding the credibility of the findings (especially in settings where the estimates are not based on randomized experiments), as well as their external validity and any evidence of heterogeneity. Kohavi, Henne, and Sommerfield (2007), Kohavi, Tang, and Xu (2020), and Gupta et al (2019) discuss in more detail the process of decision making in the context of randomized experiments in a business setting. Kohavi views experiments in this setting, and data-driven decision making more generally, as helping reduce the importance of what he has called the Highest Paid Person's Opinion (HIPPO) in less formal versions of these discussions.…”

Online Experimentation with Surrogate Metrics: Guidelines and a Case Study

“…Experiment practitioners in many elds leverage the theory and conduct experiments to evaluate new ideas [7,27]. In the technology industry, experimentation is adopted by many companies [2,12,13,19,25,29]. Deployment and analysis of controlled experiments are done at large scale.…”

“…A/B tests (or controlled experiments, split tests) have been widely adopted in the online world as the golden rule for decision making and the driving forces for innovation. Many technology companies, such as Microsoft, Google, Facebook, LinkedIn, Uber, Netix and Twitter, have in-house experimentation platforms, where experiments are run at large scale with marginal costs [2,10,12,19,25,[27][28][29]. From user-interface changes to back-end algorithms and infrastructure, from software developers to product managers to data scientists, A/B tests help make data-driven decisions and innovate new product ideas.…”