Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale

Fabijan, Aleksander; Dmitriev, Pavel; Olsson, Helena Holmström; Bosch, Jan; Vermeer, Lukas; Lewis, D.J.

doi:10.1109/icse-seip.2019.00009

Cited by 19 publications

(11 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, this project has proven the feasibility and usefulness of building data quality checks on top of existing third-party experimentation platforms when those platforms do not already perform such checks. We could extend this idea beyond SRM checks and implement additional monitoring capabilities, for example taking guidance from Fabijan et al [2] or Perrin [7].…”

Section: Discussionmentioning

confidence: 99%

“…We also want our experiments to be trustworthy, without creating knowledge or process bottlenecks. Although we will partially rely on checklists such as those suggested by Fabijan et al [2] as well as a structured education curriculum, we are also investing in automation and infrastructure to improve consistency and reliability of our data quality checks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automated Sample Ratio Mismatch (SRM) detection and analysis

Vermeer¹,

Anderson

Acebal³

2022

The International Conference on Evaluation and Assessment in Software Engineering 2022

Self Cite

View full text Add to dashboard Cite

Background: Sample Ratio Mismatch (SRM) checks can help detect data quality issues in online experimentation [3]. Not all experimentation platforms provide these checks as part of their solution. Users of these platforms must therefore manually check for SRM, or rely on additional processes-such as checklists [2]-or automation. Objective: To ensure reliable and early detection of SRM, we wanted to automate the detection and analysis of SRM in experiments running on third-party experimentation platforms. Method: A set of Looker dashboards were built to facilitate selfserve SRM detection and root cause analysis. In addition, we added email and chat based alerting to pro-actively inform experimenters of SRM and guide them towards these dashboards when needed. Results: Several cases of SRM have been detected and experimenters have been warned. Bad decisions based on flawed data were avoided. We provide one such example as an illustration. Conclusions: SRM checks are relatively straightforward to automate and can be useful for data quality monitoring even for companies who rely on third-party experimentation platforms. Pro-active alerting-rather than passive reporting-can reduce time to detection and help non-experts avoid making decisions based on biased data. CCS CONCEPTS• General and reference → Experimentation.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Automated Sample Ratio Mismatch (SRM) detection and analysis

Vermeer¹,

Anderson

Acebal³

2022

The International Conference on Evaluation and Assessment in Software Engineering 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…The trustworthiness aspect of online experiments has been an active area of research [8,14,21,24,42]. Experiments that rely on violated assumptions or are susceptible to implementation or other design errors can lead to untrustworthy results that can compromise the conclusions and the value of the experiment.…”

Section: Experimentation Processes and Platformsmentioning

confidence: 99%

“…Kohavi et al [24] discuss lessons learned from online controlled experiments that can influence the experiment result, such as carryover effects, experiment duration, and statistical power. Fabijan et al [14] provide essential checklists to prevent companies from overlooking critical trustworthiness aspects of online experiments. In our work, we do not specifically focus on trustworthiness aspects of online experiments, but on how to make the experimentation process science-centric.…”

Section: Experimentation Processes and Platformsmentioning

confidence: 99%

Engineering for a Science-Centric Experimentation Platform

Diamantopoulos¹,

Wong²,

Mattos³

et al. 2019

Preprint

View full text Add to dashboard Cite

Netflix is an internet entertainment service that routinely employs experimentation to guide strategy around product innovations. As Netflix grew, it had the opportunity to explore increasingly specialized improvements to its service, which generated demand for deeper analyses supported by richer metrics and powered by more diverse statistical methodologies. To facilitate this, and more fully harness the skill sets of both engineering and data science, Netflix engineers created a science-centric experimentation platform that leverages the expertise of data scientists from a wide range of backgrounds by allowing them to make direct code contributions in the languages used by scientists (Python and R). Moreover, the same code that runs in production is able to be run locally, making it straightforward to explore and graduate both metrics and causal inference methodologies directly into production services.In this paper, we utilize a case-study research method to provide two main contributions. Firstly, we report on the architecture of this platform, with a special emphasis on its novel aspects: how it supports science-centric end-to-end workflows without compromising engineering requirements. Secondly, we describe its approach to causal inference, which leverages the potential outcomes conceptual framework to provide a unified abstraction layer for arbitrary statistical models and methodologies.

show abstract

“…Apart from lessons learned and common pitfalls reported by experts running experiments (e.g. [5,14]), could we establish an experiment improvement process within a specific company?…”

Section: Synergies Between Disciplinesmentioning

confidence: 99%

Continuous Data-driven Software Engineering - Towards a Research Agenda

Gerostathopoulos

Konersmann

Krusche

et al. 2019

SIGSOFT Softw. Eng. Notes

View full text Add to dashboard Cite

The rapid pace with which software needs to be built, together with the increasing need to evaluate changes for end users both quantitatively and qualitatively calls for novel software engineering approaches that focus on short release cycles, continuous deployment and delivery, experiment-driven feature development, feedback from users, and rapid tool-assisted feedback to developers. To realize these approaches there is a need for research and innovation with respect to automation and tooling, and furthermore for research into the organizational changes that support flexible data-driven decision-making in the development lifecycle. Most importantly, deep synergies are needed between software engineers, managers, and data scientists. This paper reports on the results of the joint 5th International Workshop on Rapid Continuous Software Engineering (RCoSE 2019) and the 1st International Workshop on Data-Driven Decisions, Experimentation and Evolution (DDrEE 2019), which focuses on the challenges and potential solutions in the area of continuous data-driven software engineering.

show abstract

Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale

Cited by 19 publications

References 33 publications

Automated Sample Ratio Mismatch (SRM) detection and analysis

Automated Sample Ratio Mismatch (SRM) detection and analysis

Engineering for a Science-Centric Experimentation Platform

Continuous Data-driven Software Engineering - Towards a Research Agenda

Contact Info

Product

Resources

About