Trustworthy Experimentation Under Telemetry Loss

Abstract

Failure to accurately measure the outcomes of an experiment can lead to bias and incorrect conclusions. Online controlled experiments (aka AB tests) are increasingly being used to make decisions to improve websites as well as mobile and desktop applications. We argue that loss of telemetry data (during upload or post-processing) can skew the results of experiments, leading to loss of statistical power and inaccurate or erroneous conclusions. By systematically investigating the causes of telemetry loss, we argue that it is not practical to entirely eliminate it. Consequently, experimentation systems need to be robust to its effects. Furthermore, we note that it is nontrivial to measure the absolute level of telemetry loss in an experimentation system. In this paper, we take a top-down approach towards solving this problem. We motivate the impact of loss qualitatively using experiments in real applications deployed at scale, and formalize the problem by presenting a theoretical breakdown of the bias introduced by loss. Based on this foundation, we present a general framework for quantitatively evaluating the impact of telemetry loss, and present two solutions to measure the absolute levels of loss. This framework is used by well-known applications at Microsoft, with millions of users and billions of sessions. These general principles can be adopted by any application to improve the overall trustworthiness of experimentation and data-driven decision making.

Controlled Experimentation in Continuous Experimentation: Knowledge and Challenges

“…The challenges with experimentation motivate improved statistical techniques specialized for A/B testing. There are many techniques for fixing specific biases, sources of noise, etc: a specialized test for count data at SweetIM [36]; fixing errors with dependent data at Facebook [106]; improvements from the capabilities of A/A testing on diagnosis (which tests control vs control expecting no effect) at Yahoo [107] and Oath [108]; better calculation of overall effect for features with low coverage at Microsoft [109]; fixing errors from personalization interference at Yahoo [110]; fixing tests under telemetry loss at Microsoft [111]; correcting for selection bias at Airbnb [112]; and algorithms for improved gradual ramp-up at Google [113] and LinkedIn [114].…”