A Structured Multiarmed Bandit Problem and the Greedy Policy

Abstract

We consider a multiarmed bandit problem where the expected reward of each arm is a linear function of an unknown scalar with a prior distribution. The objective is to choose a sequence of arms that maximizes the expected total (or discounted total) reward. We demonstrate the effectiveness of a greedy policy that takes advantage of the known statistical correlation structure among the arms. In the infinite horizon discounted reward setting, we show that the greedy and optimal policies eventually coincide, and both settle on the best arm. This is in contrast with the Incomplete Learning Theorem for the case of independent arms. In the total reward setting, we show that the cumulative Bayes risk after periods under the greedy policy is at most (log), which is smaller than the lower bound of (log 2) established by Lai [1] for a general, but different, class of bandit problems. We also establish the tightness of our bounds. Theoretical and numerical results show that the performance of our policy scales independently of the number of arms. Index Terms-Markov decision process (MDP). I. INTRODUCTION I N the multiarmed bandit problem, a decision-maker samples sequentially from a set of arms whose reward characteristics are unknown to the decision-maker. The distribution of the reward of each arm is learned from accumulated experience as the decision-maker seeks to maximize the expected total (or discounted total) reward over a horizon. The problem has garnered significant attention as a prototypical example of the so-called exploration versus exploitation dilemma, where a decision-maker balances the incentive to exploit the arm with the highest expected payoff with the incentive to explore poorly understood arms for information-gathering purposes.

Learning to Play K-Armed Bandit Problems

“…Finally, we suggest also to extend our policy learning scheme to other -and more complexexploration-exploitation problems than the one tackled in this paper, such as for example bandit problems where the arms are not statistically independent (Mersereau et al, 2009) or general Markov Decision processes (Ishii et al, 2002).…”

Learning in Combinatorial Optimization: What and How to Explore

“…Agrawal (1995) studies a multi-armed bandit in which arms represent points in the real line and their expected rewards are continuous functions of the arms. Mersereau et al (2009) and Rusmevichientong and Tsitsiklis (2010) study bandits with possibly infinite arms when expected rewards are linear functions of an (unknown) scalar and a vector, respectively. Our paper also relates to the literature on linear bandit models (see e.g., Abernethy et al (2008) and Dani et al (2008)) as the model we study is a linear stochastic bandit with a finite (but combinatorial) number of arms.…”

The Value of Field Experiments

“…Recent work has proposed an alternative model in which the arms have statistically dependent reward distributions, and therefore pulling one arm also gives information about other arms. In this setting, the correlation between payoffs of different arms allows for faster learning, even when the number of arms is very large (Dani et al 2008, Mersereau et al 2009). …”