One-Armed Bandit Problems with Covariates

Sarkar, Jyotirmoy

doi:10.1214/aos/1176348382

Cited by 48 publications

(38 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This is the first such algorithm for adapting exploration on-line. Previous research in bandit problems in general [2], [12], and for the one-armed bandit with covariates problem in particular [9], [13], has focused on finding policies for selecting arms that converge to optimal behaviour asymptotically, and not for designing policies that autonomously maximise reward in finite time by adapting to the type of problem faced. Moreover, -ADAPT can be generalised to control exploration in a variety of bandit problems -including problems with multiple arms and dynamic problems with changing reward structures, we discuss this aspect more in Section V.…”

Section: Introductionmentioning

confidence: 99%

On-Line Adaptation of Exploration in the One-Armed Bandit with Covariates Problem

Sykulski

Adams

Jennings

2010

2010 Ninth International Conference on Machine Learning and Applications

View full text Add to dashboard Cite

Abstract-Many sequential decision making problems require an agent to balance exploration and exploitation to maximise long-term reward. Existing policies that address this tradeoff typically have parameters that are set a priori to control the amount of exploration. In finite-time problems, the optimal values of these parameters are highly dependent on the problem faced. In this paper, we propose adapting the amount of exploration performed on-line, as information is gathered by the agent. To this end we introduce a novel algorithm, -ADAPT, which has no free parameters. The algorithm adapts as it plays and sequentially chooses whether to explore or exploit, driven by the amount of uncertainty in the system. We provide simulation results for the onearmed bandit with covariates problem, which demonstrate the effectiveness of -ADAPT to correctly control the amount of exploration in finite-time problems and yield rewards that are close to optimally tuned off-line policies. Furthermore, we show that -ADAPT is robust to a high-dimensional covariate, as well as misspecified models. Finally, we describe how our methods could be extended to other sequential decision making problems, such as dynamic bandit problems with changing reward structures.

show abstract

Section: Introductionmentioning

confidence: 99%

On-Line Adaptation of Exploration in the One-Armed Bandit with Covariates Problem

Sykulski

Adams

Jennings

2010

2010 Ninth International Conference on Machine Learning and Applications

View full text Add to dashboard Cite

show abstract

“…The second inequality follows from the condition that the second coordinate of the estimate, ϑ = θ 2 , and then extending the finite sum to the infinite sum. The third inequality follows from the definition of Cond3a and changing the time index to τ ′ , similarly to the reasoning in (19)- (20). By Sanov's theorem, each term is exponentially upper bounded w.r.t.…”

Section: Appendix V Proof Of Theoremmentioning

confidence: 83%

“…By Sanov's theorem on R (Theorem 9), the probability of each term in (20) is exponentially upper bounded w.r.t. τ ′ , which implies that the summation has bounded expectation.…”

Section: Appendix V Proof Of Theoremmentioning

confidence: 99%

“…The extent to which this side observation can help depends on the relationship of X t to the reward distributions of Y 1 t and Y 2 t . Previous work on bandit problems with side observations includes [18], [19], [20], [21], [22]. Woodroofe [21] considered a one-armed bandit in a Bayesian setting, and constructed a simple criterion for asymptotically optimal rules.…”

mentioning

confidence: 99%

“…Woodroofe [21] considered a one-armed bandit in a Bayesian setting, and constructed a simple criterion for asymptotically optimal rules. Sarkar [20] extended the side information model of [21] to the exponential family. In [19], Kulkarni considered classes of reward distributions and their effects on performance using results from learning theory.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Bandit problems with side observations

Wang

Kulkarni

Poor

Proceedings of the 41st IEEE Conference on Decision and Control, 2002.

View full text Add to dashboard Cite

Abstract-An extension of the traditional two-armed bandit problem is considered, in which the decision maker has access to some side information before deciding which arm to pull. At each time t, before making a selection, the decision maker is able to observe a random variable Xt that provides some information on the rewards to be obtained. The focus is on finding uniformly good rules (that minimize the growth rate of the inferior sampling time) and on quantifying how much the additional information helps. Various settings are considered and for each setting, lower bounds on the achievable inferior sampling time are developed and asymptotically optimal adaptive schemes achieving these lower bounds are constructed.

show abstract