Bandit problems with infinitely many arms

Berry, Donald A.; Chen, Robert W.; Zame, Alan; Heath, David; Shepp, Larry

doi:10.1214/aos/1069362389

Cited by 112 publications

(77 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A strong probabilistic assumption that has been made in [17,19] to model such situations is that the mean-value of any unobserved arm is a random variable that follows some known distribution. More recently this assumption has been weakened in [108] with an assumption focussing on this distribution upper tail only.…”

Section: Unstructured Rewardsmentioning

confidence: 99%

From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning

Munos

2014

FNT in Machine Learning

192

181

View full text Add to dashboard Cite

This work covers several aspects of the optimism in the face of uncertainty principle applied to large scale optimization problems under finite numerical budget. The initial motivation for the research reported here originated from the empirical success of the so-called Monte-Carlo Tree Search method popularized in Computer Go and further extended to many other games as well as optimization and planning problems. Our objective is to contribute to the development of theoretical foundations of the field by characterizing the complexity of the underlying optimization problems and designing efficient algorithms with performance guarantees. The main idea presented here is that it is possible to decompose a complex decision making problem (such as an optimization problem in a large search space) into a sequence of elementary decisions, where each decision of the sequence is solved using a (stochastic) multi-armed bandit (simple mathematical model for decision making in stochastic environments). This so-called hierarchical bandit approach (where the reward observed by a bandit in the hierarchy is itself the return of another bandit at a deeper level) possesses the nice feature of starting the exploration by a quasi-uniform sampling of the space and then focusing progressively on the most promising area, at different scales, according to the evaluations observed so far, and eventually performing a local search around the global optima of the function. The performance of the method is assessed in terms of the optimality of the returned solution as a function of the number of function evaluations. Our main contribution to the field of function optimization is a class of hierarchical optimistic algorithms designed for general search spaces (such as metric spaces, trees, graphs, Euclidean spaces, ...) with different algorithmic instantiations depending on whether the evaluations are noisy or noiseless and whether some measure of the "smoothness" of the function is known or unknown. The performance of the algorithms depends on the local behavior of the function around its global optima expressed in terms of the quantity of near-optimal states measured with some metric. If this local smoothness of the function is known then one can design very efficient optimization algorithms (with convergence rate independent of the space dimension), and when it is not known, we can build adaptive techniques that can, in some cases, perform almost as well as when it is known. In order to be self-contained, we start with a brief introduction to the stochastic multi-armed bandit problem in Chapter 1 and describe the UCB (Upper Confidence Bound) strategy and several extensions. In Chapter 2 we present the Monte-Carlo Tree Search method applied to Computer Go and show the limitations of previous algorithms such as UCT (UCB applied to Trees). This provides motivation for designing theoretically well-founded optimistic optimization algorithms. The main contributions on hierarchical optimistic optimization are described in Chapters 3 and 4...

show abstract

Section: Unstructured Rewardsmentioning

confidence: 99%

From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning

Munos

2014

FNT in Machine Learning

192

181

View full text Add to dashboard Cite

show abstract

“…Examples include Berry et al (1997) who consider a Bayesian setting with infinite Bernoulli arms, and Kleinberg (2004Kleinberg ( , 2005 who consider applications to network routing and single-product pricing. However, most of these works are very problem-specific, and they are not applicable to our assortment optimization problem.…”

Section: Connections To Multiarmed Bandit Problemsmentioning

confidence: 99%

Dynamic Assortment Optimization with a Multinomial Logit Choice Model and Capacity Constraint

Rusmevichientong

Shen

Shmoys

2010

Operations Research

422

240

View full text Add to dashboard Cite

The paper considers a stylized model of a dynamic assortment optimization problem, where given a limited capacity constraint, we must decide the assortment of products to offer to customers to maximize the profit. Our model is motivated by the problem faced by retailers of stocking products on a shelf with limited capacities and by the problem of placing a limited number of ads on a web page. We assume that each customer chooses to purchase the product (or to click on the ad) that maximizes her utility. We use the multinomial logit choice model to represent demand. However, we do not know the demand for each product. We can learn the demand distribution by offering different product assortments, observing resulting selections, and inferring the demand distribution from past selections and assortment decisions. We present an adaptive policy for joint parameter estimation and assortment optimization. To evaluate our proposed policy, we define a benchmark profit as the maximum expected profit that we can earn if we know the underlying demand distribution in advance. We show that the running average expected profit generated by our policy converges to the benchmark profit and establish its convergence rate. Numerical experiments based on sales data from an online retailer indicate that our policy performs well, generating over 90% of the optimal profit after less than two days of sales.

show abstract

“…The infinite bandit problem [1,2] and to the secretary problem [7] are the closest formulations we know of, in that they require some form of stopping, but differ in aspects i) and ii).…”

Section: Related Workmentioning

confidence: 99%

Greedy-Bayes for Targeted News Dissemination

Massoulié

Ohannessian

Proutière

2015

Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems

View full text Add to dashboard Cite

International audienceThis work addresses user targeting for news content delivery. Specifically, we wish to disseminate a fresh news content, whose topic is yet unknown, to all interested users, while " spamming " a minimum number of uninterested users. We formulate this as an online stochastic optimization problem that extends in several ways the classical multi-armed bandit problem. We introduce Greedy-Bayes, a policy with appealing ro-bustness properties. We establish optimal scaling of a suitably defined regret measure in various scenarios of interest. To that end we develop an original proof technique based on martingale concentration inequalities. Numerical experiments show that Greedy-Bayes improves upon Thompson sampling, the state-of-the-art algorithm for bandit problems. Our analysis further implies that low regret can only be achieved if the assessment of content relevance for one user leverages feedback from users with widely distinct tastes. This impacts the design of efficient news dissemination platforms: existing systems typically do not leverage such negative feedback and could hence be improved upon with adequate extensions

show abstract

Bandit problems with infinitely many arms

Cited by 112 publications

References 7 publications

From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning

From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning

Dynamic Assortment Optimization with a Multinomial Logit Choice Model and Capacity Constraint

Greedy-Bayes for Targeted News Dissemination

Contact Info

Product

Resources

About