Choosing a Good Toolkit, II: Simulations and Conclusions

Francetich, Alejandro; Kreps, David M.

doi:10.2139/ssrn.2813463

Cited by 3 publications

(3 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A notable exception is the KG algorithm, which takes a discount factor as input to account for time preference. Francetich and Kreps [26,27] discuss a variety of heuristics for the discounted problem. Recent work [62] generalizes Thompson sampling to address discounted problems.…”

Section: Literature Reviewmentioning

confidence: 99%

Learning to Optimize via Information-Directed Sampling

Russo

Roy

2018

Operations Research

110

207

View full text Add to dashboard Cite

Abstract. We propose information-directed sampling-a new approach to online optimization problems in which a decision maker must balance between exploration and exploitation while learning from partial feedback. Each action is sampled in a manner that minimizes the ratio between squared expected single-period regret and a measure of information gain: the mutual information between the optimal action and the next observation.We establish an expected regret bound for information-directed sampling that applies across a very general class of models and scales with the entropy of the optimal action distribution. We illustrate through simple analytic examples how information-directed sampling accounts for kinds of information that alternative approaches do not adequately address and that this can lead to dramatic performance gains. For the widely studied Bernoulli, Gaussian, and linear bandit problems, we demonstrate state-of-the-art simulation performance.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Learning to Optimize via Information-Directed Sampling

Russo

Roy

2018

Operations Research

110

207

View full text Add to dashboard Cite

show abstract

“…Our discounted framework instead focuses on the initial cost of learning to attain good, but not perfect, performance. Recent papers [9,10] study several heuristics for a discounted objective, though without an orientation toward formal regret analysis. The Knowledge Gradient algorithm of [15] also takes time horizon into account and can learn suboptimal actions when its not worthwhile to identify the optimal action.…”

Section: Introductionmentioning

confidence: 99%

Time-Sensitive Bandit Learning and Satisficing Thompson Sampling

Russo¹,

Tse²,

Roy³

2017

Preprint

View full text Add to dashboard Cite

The literature on bandit learning and regret analysis has focused on contexts where the goal is to converge on an optimal action in a manner that limits exploration costs. One shortcoming imposed by this orientation is that it does not treat time preference in a coherent manner. Time preference plays an important role when the optimal action is costly to learn relative to near-optimal actions. This limitation has not only restricted the relevance of theoretical results but has also influenced the design of algorithms. Indeed, popular approaches such as Thompson sampling and UCB can fare poorly in such situations. In this paper, we consider discounted rather than cumulative regret, where a discount factor encodes time preference. We propose satisficing Thompson sampling -a variation of Thompson sampling -and establish a strong discounted regret bound for this new algorithm.

show abstract

“…Our discounted framework instead focuses on the initial cost of learning to attain good, but not perfect, performance. Recent papers [11,12] study several heuristics for a discounted objective, though without an orientation toward formal regret analysis.…”

Section: Alternative Approachesmentioning

confidence: 99%

Satisficing in Time-Sensitive Bandit Learning

Russo¹,

Roy²

2018

Preprint

View full text Add to dashboard Cite

Much of the recent literature on bandit learning focuses on algorithms that aim to converge on an optimal action. One shortcoming is that this orientation does not account for time sensitivity, which can play a crucial role when learning an optimal action requires much more information than near-optimal ones. Indeed, popular approaches such as upper-confidence-bound methods and Thompson sampling can fare poorly in such situations. We consider instead learning a satisficing action, which is near-optimal while requiring less information, and propose satisficing Thompson sampling, an algorithm that serves this purpose. We establish a general bound on expected discounted regret and study the application of satisficing Thompson sampling to linear and infinite-armed bandits, demonstrating arbitrarily large benefits over Thompson sampling. We also discuss the relation between the notion of satisficing and the theory of rate distortion, which offers guidance on the selection of satisficing actions.

show abstract

Choosing a Good Toolkit, II: Simulations and Conclusions

Cited by 3 publications

References 1 publication

Learning to Optimize via Information-Directed Sampling

Learning to Optimize via Information-Directed Sampling

Time-Sensitive Bandit Learning and Satisficing Thompson Sampling

Satisficing in Time-Sensitive Bandit Learning

Contact Info

Product

Resources

About