Engineering for a Science-Centric Experimentation Platform

Diamantopoulos, Nikos; Wong, Jeffrey Y.C.; Mattos, David Issa; Gerostathopoulos, Ilias; Wardrop, Matthew; Mao, Tobias; McFarland, Colin

doi:10.48550/arxiv.1910.03878

Cited by 3 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experimentation is ubiquitous in online services such as Facebook, LinkedIn [29], Netflix [9], etc., where the effects of product changes are explicitly tested and analyzed in randomized trials. Interference, sometimes referred to as network effects in the context of online social networks, is a threat to the validity of these randomized trials as the presence of interference violates the stable unit treatment value assumption (SUTVA, see e.g.…”

Section: Introductionmentioning

confidence: 99%

Network Experimentation at Scale

Karrer

Shi

Bhole

et al. 2021

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

We describe our framework, deployed at Facebook, that accounts for interference between experimental units through cluster-randomized experiments. We document this system, including the design and estimation procedures, and detail insights we have gained from the many experiments that have used this system at scale. We introduce a cluster-based regression adjustment that substantially improves precision for estimating global treatment effects as well as testing for interference as part of our estimation procedure. With this regression adjustment, we find that imbalanced clusters can better account for interference than balanced clusters without sacrificing accuracy. In addition, we show how logging exposure to a treatment can be used for additional variance reduction. Interference is a widely acknowledged issue with online field experiments, yet there is less evidence from real-world experiments demonstrating interference in online settings. We fill this gap by describing two case studies that capture significant network effects and highlight the value of this experimentation framework.

show abstract

Section: Introductionmentioning

confidence: 99%

Network Experimentation at Scale

Karrer

Shi

Bhole

et al. 2021

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

show abstract

“…A/B tests are widely used as the gold standard for understanding how a new algorithm will behave at scale. Almost all large tech companies routinely use A/B tests to evaluate changes before deploying them [18,22,34,48,53,58,63,70,76]. Networking research often includes the results of A/B tests, and uses them to justify new algorithms [17-19, 24, 25, 29, 42, 46, 49, 55, 57, 58, 60, 63, 72, 87].…”

Section: Introductionmentioning

confidence: 99%

Unbiased experiments in congested networks

Spang

Hannan

Kunamalla

et al. 2021

Proceedings of the 21st ACM Internet Measurement Conference

View full text Add to dashboard Cite

When developing a new networking algorithm, it is established practice to run a randomized experiment, or A/B test, to evaluate its performance. In an A/B test, traffic is randomly allocated between a treatment group, which uses the new algorithm, and a control group, which uses the existing algorithm. However, because networks are congested, both treatment and control traffic compete against each other for resources in a way that biases the outcome of these tests. This bias can have a surprisingly large effect; for example, in lab A/B tests with two widely used congestion control algorithms, the treatment appeared to deliver 150% higher throughput when used by a few flows, and 75% lower throughput when used by most flowsdespite the fact that the two algorithms have identical throughput when used by all traffic.Beyond the lab, we show that A/B tests can also be biased at scale. In an experiment run in cooperation with Netflix, estimates from A/B tests mistake the direction of change of some metrics, miss changes in other metrics, and overestimate the size of effects. We propose alternative experiment designs, previously used in online platforms, to more accurately evaluate new algorithms and allow experimenters to better understand the impact of congestion on their tests.

show abstract

“…It is common to find mature software and engineering systems for an experimentation platform (Fabijan et al 2017, Kohavi et al 2013, Deng, Lu, and Litz 2017, Diamantopoulos et al 2019). However, it is much less common to find mature software for statistics and causal inference that integrates into such systems.…”

Section: Introductionmentioning

confidence: 99%

Computational Causal Inference

Wong¹

2020

Preprint

View full text Add to dashboard Cite

We introduce computational causal inference as an interdisciplinary field across causal inference, algorithms design and numerical computing. The field aims to develop software specializing in causal inference that can analyze massive datasets with a variety of causal effects, in a performant, general, and robust way. The focus on software improves research agility, and enables causal inference to be easily integrated into large engineering systems. In particular, we use computational causal inference to deepen the relationship between causal inference, online experimentation, and algorithmic decision making.This paper describes the new field, the demand, opportunities for scalability, open challenges, and begins the discussion for how the community can unite to solve challenges for scaling causal inference and decision making.

show abstract

Engineering for a Science-Centric Experimentation Platform

Cited by 3 publications

References 22 publications

Network Experimentation at Scale

Network Experimentation at Scale

Unbiased experiments in congested networks

Computational Causal Inference

Contact Info

Product

Resources

About