An Efficient Bandit Algorithm for Realtime Multivariate Optimization

Abstract

Optimization is commonly employed to determine the content of web pages, such as to maximize conversions on landing pages or click-through rates on search engine result pages. Often the layout of these pages can be decoupled into several separate decisions. For example, the composition of a landing page may involve deciding which image to show, which wording to use, what color background to display, etc. Such optimization is a combinatorial problem over an exponentially large decision space. Randomized experiments do not scale well to this setting, and therefore, in practice, one is typically limited to optimizing a single aspect of a web page at a time. This represents a missed opportunity in both the speed of experimentation and the exploitation of possible interactions between layout decisions.Here we focus on multivariate optimization of interactive web pages. We formulate an approach where the possible interactions between different components of the page are modeled explicitly. We apply bandit methodology to explore the layout space efficiently and use hill-climbing to select optimal content in realtime. Our algorithm also extends to contextualization and personalization of layout selection. Simulation results show the suitability of our approach to large decision spaces with strong interactions between content. We further apply our algorithm to optimize a message that promotes adoption of an Amazon service. After only a single week of online optimization, we saw a 21% conversion increase compared to the median layout. Our technique is currently being deployed to optimize content across several locations at Amazon.com.

A Batched Multi-Armed Bandit Approach to News Headline Testing

“…That is, the success probability θ of a headline variant may be influenced by the titles of the remaining articles in the same module. This motivates us to extend our research and develop an MAB testing framework that optimizes the article titles for the entire module, e.g., with a multivariate MAB algorithm introduced in [20].…”

“…[11] also mentions Thompson Sampling can be implemented efficiently, in comparison with full Bayesian methods such as Gittins index. Recently, adaptations of Thompson Sampling have been applied in many domains, such as revenue management [24], recommendation system [25], online service experiments [19], website optimization [20], and online advertising [16] [17] [18].…”

“…Thompson Sampling with batch updates had been studied in display advertising and news article recommendation to analyze how it performs in the case of delayed user response processing compared with other MAB algorithms [11]. Batch update is incorporated in recent industrial applications of Thompson Sampling [20] [21]. Although how they select the update frequency is not disclosed, [20] adopts a method to update traffic allocation once a day, and [21] updates twice a day.…”

“…Batch update is incorporated in recent industrial applications of Thompson Sampling [20] [21]. Although how they select the update frequency is not disclosed, [20] adopts a method to update traffic allocation once a day, and [21] updates twice a day.…”

Seeker

“…To keep our notation consistent, we always assume that the user prefers item x i to x j when we write s i j ∈ S. We make the further assumption that each preference pair is independent from each other. This is a simplifying assumption which serves as a good baseline [6,12]. In Section 6.1, we investigate ways to drop the independence assumption.…”