Online Learning for Non-Stationary A/B Tests

Abstract

The rollout of new versions of a feature in modern applications is a manual multi-stage process, as the feature is released to ever larger groups of users, while its performance is carefully monitored. This kind of A/B testing is ubiquitous, but suboptimal, as the monitoring requires heavy human intervention, is not guaranteed to capture consistent, but short-term fluctuations in performance, and is inefficient, as better versions take a long time to reach the full population. In this work we formulate this question as that of expert learning, and give a new algorithm Follow-The-Best-Interval, FTBI, that works in dynamic, non-stationary environments. Our approach is practical, simple, and efficient, and has rigorous guarantees on its performance. Finally, we perform a thorough evaluation on synthetic and real world datasets and show that our approach outperforms current state-of-the-art methods. CCS CONCEPTS • Information systems → Computational advertising; • Theory of computation → Online learning algorithms; Computational pricing and auctions; • Security and privacy → Database activity monitoring;

The Technological Gap Between Virtual Assistants and Recommendation Systems

“…For example, this will be achieved by applying the Word2Vec strategy to create the textual embedding and measure their coherence based on the respective embeddings' similarities [63], as well as following collaborative topic modeling strategies [64]. A/B online testing in conversational systems has also to be performed, allowing the comparison of different conversation policies [65].…”