Optimistic Gittins Indices

Farias, Vivek F.; Gutin, Eli

doi:10.1287/opre.2021.2207

Cited by 2 publications

(1 citation statement)

References 32 publications

(31 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further works include extensions to the discounted infinite horizon case, to the frequentist setting, as well as the theoretical analysis of the DeCo policy. Also, while we have not reproduced their results, the DeCo policy seems to have performance close to the state of the art [19,9].…”

Section: Discussioncontrasting

confidence: 55%

Decomposition-Coordination Method for Finite Horizon Bandit Problems

de Lara,

Heymann,

Chancelier

2021

Preprint

View full text Add to dashboard Cite

Optimally solving a multi-armed bandit problem suffers the curse of dimensionality. Indeed, resorting to dynamic programming leads to an exponential growth of computing time, as the number of arms and the horizon increase. We introduce a decompositioncoordination heuristic, DeCo, that turns the initial problem into parallelly coordinated one-armed bandit problems. As a consequence, we obtain a computing time which is essentially linear in the number of arms. In addition, the decomposition provides a theoretical lower bound on the regret. For the two-armed bandit case, dynamic programming provides the exact solution, which is almost matched by the DeCo heuristic. Moreover, in numerical simulations with up to 100 rounds and 20 arms, DeCo outperforms classic algorithms (Thompson sampling and Kullback-Leibler upper-confidence bound) and almost matches the theoretical lower bound on the regret for 20 arms.

show abstract

Section: Discussioncontrasting

confidence: 55%

Decomposition-Coordination Method for Finite Horizon Bandit Problems

de Lara,

Heymann,

Chancelier

2021

Preprint

View full text Add to dashboard Cite

show abstract

Thompson Sampling with Information Relaxation Penalties

Min,

Maglaras,

Moallemi

2024

Management Science

View full text Add to dashboard Cite

We consider a finite-horizon multiarmed bandit (MAB) problem in a Bayesian setting, for which we propose an information relaxation sampling framework. With this framework, we define an intuitive family of control policies that include Thompson sampling (TS) and the Bayesian optimal policy as endpoints. Analogous to TS, which at each decision epoch pulls an arm that is best with respect to the randomly sampled parameters, our algorithms sample entire future reward realizations and take the corresponding best action. However, this is done in the presence of “penalties” that seek to compensate for the availability of future information. We develop several novel policies and performance bounds for MAB problems that vary in terms of improving performance and increasing computational complexity between the two endpoints. Our policies can be viewed as natural generalizations of TS that simultaneously incorporate knowledge of the time horizon and explicitly consider the exploration-exploitation trade-off. We prove associated structural results on performance bounds and suboptimality gaps. Numerical experiments suggest that this new class of policies perform well, in particular, in settings where the finite time horizon introduces significant exploration-exploitation tension into the problem. Finally, inspired by the finite-horizon Gittins index, we propose an index policy that builds on our framework that particularly outperforms the state-of-the-art algorithms in our numerical experiments. This paper was accepted by Hamid Nazerzadeh, data science. Funding: This research was supported by the National Research Foundation of Korea [NRF-2022R1C1C1013402]. Supplemental Material: The electronic companion and data files are available at https://doi.org/10.1287/mnsc.2020.01396 .

show abstract

Optimistic Gittins Indices

Cited by 2 publications

References 32 publications

Decomposition-Coordination Method for Finite Horizon Bandit Problems

Decomposition-Coordination Method for Finite Horizon Bandit Problems

Thompson Sampling with Information Relaxation Penalties

Contact Info

Product

Resources

About