On the Gittins Index for Multiarmed Bandits

Weber, Richard

doi:10.1214/aoap/1177005588

Cited by 220 publications

(112 citation statements)

References 0 publications

Supporting

Mentioning

111

Contrasting

Order By: Relevance

“…This gives rise to new interpretations of the indices as prices or retirement options. For example, we can obtain a new interpretation of indices in the context of branching bandits as retirement options, thus generalizing the interpretation of Whittle [61] and Weber [59] for the indices of the classical multiarmed bandit problem.…”

mentioning

confidence: 83%

See 1 more Smart Citation

The achievable region method in the optimal control of queueing systems; formulations, bounds and policies

Bertsimas

1995

Queueing Syst

View full text Add to dashboard Cite

We survey a new approach that the author and his co-workers have developed to formulate stochastic control problems (predominantly queueing systems) as mathematicalprogramming problems. The central idea is to characterize the region of achievable performance in a stochastic control problem, i.e., find linear or nonlinear constraints on the performance vectors that all policies satisfy. We present linear and nonlinear relaxations of the performance space for the following problems: Indexable systems (multiclass single station queues and multiarmed bandit problems), restless bandit problems, polling systems, multiclass queueing and loss networks. These relaxations lead to bounds on the performance of an optimal policy. Using information from the relaxations we construct heuristic nearly optimal policies, The theme in the paper is the thesis that better formulations lead to deeper understanding and better solution methods. Overall the proposed approach for stochastic control problems parallels efforts of the mathematical programming community in the last twenty years to develop sharper formulations (polyhedral combinatorics and more recently nonlinear relaxations) and leads to new insights ranging from a complete characterization and new algorithms for indexable systems to tight lower bounds and nearly optimal algorithms for restless bandit problems, polling systems, multiclass queueing and loss networks.

show abstract

mentioning

confidence: 83%

“…Varaiya, Walrand and Buyukkoc [58] and Weiss [60] provided different proofs based on interchange arguments. Weber [59] and Tsitsiklis [56] outlined intuitive proofs.…”

Section: Indexable Systemsmentioning

confidence: 99%

The achievable region method in the optimal control of queueing systems; formulations, bounds and policies

Bertsimas

1995

Queueing Syst

View full text Add to dashboard Cite

show abstract

“…The purpose of this presentation is to clarify the underlying properties of the bandit problem that make such a simple policy optimal. Frostig and Weiss (1999) surveyed four proofs for the optimality of the Gittins index in the literature: Gittins' original proof of interchange argument (Gittins (1979)), Weber's fair charge argument (Weber (1992)), Whittle's dual Lagrangian approach (Whittle (1980)), and Bertsimas and Niño-Mora's linear programming approach of achievable region and generalized conservation laws (Bertsimas and Niño-Mora (1996)). Although these proofs are different in the details, they share a common strategy: the proof starts with a study of a single-armed bandit process, and the optimal policy is solved for in this restricted case.…”

Section: No Switching Costsmentioning

confidence: 99%

“…To name a few example, a worker who switches jobs must pay non-negligible costs. 18 16 Proof by Weber (1992) can be translated into Whittle (1980)'s proof by setting fair charge γ equal to (1 − β)M. 17 This property also holds when switching arms is costly. I will discuss this in detail in Asawa and Teneketzis (1996) in Section 5.…”

Section: Switching Costsmentioning

confidence: 99%

A survey on the bandit problem with switching costs

Jun

2004

De Economist

View full text Add to dashboard Cite

SummaryThe paper surveys the literature on the bandit problem, focusing on its recent development in the presence of switching costs. Switching costs between arms makes not only the Gittins index policy suboptimal, but also renders the search for the optimal policy computationally infeasible. This survey will first discuss the decomposability properties of the arms that make the Gittins index policy optimal, and show how these properties break down upon the introduction of costs on switching arms. Having established the failure of the simple index policy, the survey focus on the recent efforts to overcome the difficulty of finding the optimal policy in the bandit problem with switching costs: characterization of the optimal policy, exact derivation of the optimal policy in the restricted environments, and lastly approximation of optimal policy. The advantages and disadvantages of the above approaches are discussed.

show abstract

“…An extensive literature exists outlining a range of extensions and developments of Gittins' classical work while various schemes for index computation have been proposed. See, for example, Whittle (1980), Weber (1992), Katehakis and Veinott (1987) and Bertsimas and Niño-Mora (1996).…”

Section: Introductionmentioning

confidence: 99%

Index Policies for Shooting Problems

Glazebrook

Kirkbride

Mitchell

et al. 2007

Operations Research

View full text Add to dashboard Cite

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, ABSTRACT (maximum 200 words)We consider a scenario in which a single Red wishes to shoot at a collection of Blue targets, one at a time, to maximise some measure of return obtained from Blues killed before Red's own (possible) demise. Such a situation arises in various military contexts such as the conduct of air defence by Red in the face of Blue SEAD (suppression of enemy air defences). A class of decision processes called multi-armed bandits has been previously deployed to develop optimal policies for Red in which she attaches a calibrating (Gittins) index to each Blue target and optimally shoots next at the Blue with largest index value. The current paper seeks to elucidate how a range of developments of index theory are able to accommodate features of such problems which are of practical military import. Such features include levels of risk to Red which are policy dependent, Red having imperfect information about the Blues she faces, an evolving population of Blue targets and the possibility of Red disengagement. The paper concludes with a numerical study which both compares the performance of (optimal) index policies to a range of competitors and also demonstrates the value to Red of (optimal) disengagement. NUMBER OF PAGES 2714. SUBJECT TERMS multi-armed bandits, Gitten Indices, suppression of enemy air defense PRICE CODE SECURITY CLASSIFICATION OF REPORT Unclassified SECURITY CLASSIFICATION OF THIS PAGE Unclassified SECURITY CLASSIFICATION OF ABSTRACT Unclassified LIMITATION OF ABSTRACT UL iiIndex policies for shooting problems AbstractWe consider a scenario in which a single Red wishes to shoot at a collection of Blue targets, one at a time, to maximise some measure of return obtained from Blues killed before Red's own (possible) demise. Such a situation arises in various military contexts such as the conduct of air defence by Red in the face of Blue SEAD (suppression of enemy air defences). A class of decision processes called multi-armed bandits has been previously deployed to develop optimal policies for Red in which she attaches a calibrating (Gittins) index to each Blue target and optimally shoots next at the Blue with largest index value. The current paper seeks to elucidate how a range of developments of index theory are able to accommodate features of such problems which are of practical military import. Such features include levels of risk to Red which are policy dependent, Red having imperfect information about the Blues she faces, an evolving population of Blue targets and the possibility of Red disengagement. The paper concludes with a numerical study which both c...

show abstract

On the Gittins Index for Multiarmed Bandits

Cited by 220 publications

References 0 publications

The achievable region method in the optimal control of queueing systems; formulations, bounds and policies

The achievable region method in the optimal control of queueing systems; formulations, bounds and policies

A survey on the bandit problem with switching costs

Index Policies for Shooting Problems

Contact Info

Product

Resources

About