Four proofs of Gittins’ multiarmed bandit theorem

Frostig, Esther; Weiss, Gideon

doi:10.1007/s10479-013-1523-0

Cited by 52 publications

(15 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proof of this result has been obtained using a number of different perspectives, for example Weber's prevailing charge formulation [75] (which we consider in more detail below), Whittle's retirement option formulation [76] and its extension without a Markov assumption by El Karoui and Karatzas [32] (and [33] in continuous time). A review of the proofs in discrete time is given by Frostig and Weiss [39]. However, in all these cases, the objective to be optimized is the discounted expected gain/loss -in particular, we are assumed to have no risk-aversion or uncertainty-aversion.…”

Section: Multi-armed Banditsmentioning

confidence: 99%

“…The subtlety in the proof of Gittins' theorem is to give a tractable representation of the class of control policies available to the decision maker. In the original formulation (see, for example, [41,75,76,39]), the class considered is feedback controls, as in a standard Markovian stochastic control problem; i.e. the system of bandits is modelled as a single Markov process, and the controls alter its transition probabilities.…”

Section: General Problem Formulationmentioning

confidence: 99%

See 1 more Smart Citation

Gittins’ theorem under uncertainty

Cohen

Treetanthiploet

2022

Electron. J. Probab.

View full text Add to dashboard Cite

We study dynamic allocation problems for discrete time multi-armed bandits under uncertainty, based on the the theory of nonlinear expectations. We show that, under independence assumption on the bandits and with some relaxation in the definition of optimality, a Gittins allocation index gives optimal choices. This involves studying the interaction of our uncertainty with controls which determine the filtration. We also run a simple numerical example which illustrates the interaction between the willingness to explore and uncertainty aversion of the agent when making decisions.

show abstract

Section: Multi-armed Banditsmentioning

confidence: 99%

Section: General Problem Formulationmentioning

confidence: 99%

Gittins’ theorem under uncertainty

Cohen

Treetanthiploet

2022

Electron. J. Probab.

View full text Add to dashboard Cite

show abstract

“…Central results are the existence and form of index-based policies for certain models that maximize the present value of expected rewards, cf. Gittins et al [19], Frostig and Weiss [18], Mahajan and Teneketzis [37], Kaspi and Mandelbaum [28], Ishikida and Varaiya [27], El Karoui and Karatzas [14], Gittins [22], Gittins [21] and Gittins et al [20].…”

Section: Introductionmentioning

confidence: 99%

Multi-Armed Bandits Under General Depreciation and Commitment

Cowan

Katehakis

2014

Prob. Eng. Inf. Sci.

View full text Add to dashboard Cite

Generally, the multi-armed has been studied under the setting that at each time step over an infinite horizon a controller chooses to activate a single process or bandit out of a finite collection of independent processes (statistical experiments, populations, etc.) for a single period, receiving a reward that is a function of the activated process, and in doing so advancing the chosen process. Classically, rewards are discounted by a constant factor β ∈ (0, 1) per round.In this paper, we present a solution to the problem, with potentially non-Markovian, uncountable state space reward processes, under a framework in which, first, the discount factors may be non-uniform and vary over time, and second, the periods of activation of each bandit may be not be fixed or uniform, subject instead to a possibly stochastic duration of activation before a change to a different bandit is allowed. The solution is based on generalized restart-in-state indices, and it utilizes a view of the problem not as "decisions over state space" but rather "decisions over time".

show abstract

“…The classical MAB problem has an optimal solution given by the Gittins index policy which associates a dynamic index to each arm and then plays the arm with the highest index in each period (see Frostig and Weiss [12] for several proofs of this result). In this section we define and analyze an index policy for the RMAB model in Equation (1).…”

Section: Robust Index Policymentioning

confidence: 99%

Robust Control of the Multi-Armed Bandit Problem

Caro

Gupta

2013

SSRN Journal

View full text Add to dashboard Cite

We study a robust model of the multi-armed bandit (MAB) problem in which the transition probabilities are ambiguous and belong to subsets of the probability simplex. We first show that for each arm there exists a robust counterpart of the Gittins index that is the solution to a robust optimal stopping-time problem and can be computed effectively with an equivalent restart problem. We then characterize the optimal policy of the robust MAB as a project-by-project retirement policy but we show that arms become dependent so the policy based on the robust Gittins index is not optimal. For a project selection problem, we show that the robust Gittins index policy is near optimal but its implementation requires more computational effort than solving a non-robust MAB problem. Hence, we propose a Lagrangian index policy that requires the same computational effort as evaluating the indices of a non-robust MAB and is within 1% of the optimum in the robust project selection problem.

show abstract

Four proofs of Gittins’ multiarmed bandit theorem

Cited by 52 publications

References 40 publications

Gittins’ theorem under uncertainty

Gittins’ theorem under uncertainty

Multi-Armed Bandits Under General Depreciation and Commitment

Robust Control of the Multi-Armed Bandit Problem

Contact Info

Product

Resources

About