The Multi-Armed Bandit Problem: Decomposition and Computation

Katehakis, Michael N.; Veinott, Arthur F.

doi:10.1287/moor.12.2.262

Cited by 271 publications

(168 citation statements)

References 5 publications

Supporting

Mentioning

167

Contrasting

Unclassified

Order By: Relevance

“…We show that the optimal policy for the multi-armed under this generalized depreciation model is an index policy, where the indices are propitiously generalized restart in state indices, cf. Katehakis and Veinott [33] and Katehakis et al [29]; see also Sonin [43], Sonin [44] and Steinberg and Sonin [45]. Furthermore, the overall proof suggests a way of understanding the structure of the reward processes, relative to a "natural" time scale of possibly stochastic intervals of activation or "restart blocks", rather than steps of unit time.…”

Section: Introductionmentioning

confidence: 81%

Multi-Armed Bandits Under General Depreciation and Commitment

Cowan

Katehakis

2014

Prob. Eng. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

Generally, the multi-armed has been studied under the setting that at each time step over an infinite horizon a controller chooses to activate a single process or bandit out of a finite collection of independent processes (statistical experiments, populations, etc.) for a single period, receiving a reward that is a function of the activated process, and in doing so advancing the chosen process. Classically, rewards are discounted by a constant factor β ∈ (0, 1) per round.In this paper, we present a solution to the problem, with potentially non-Markovian, uncountable state space reward processes, under a framework in which, first, the discount factors may be non-uniform and vary over time, and second, the periods of activation of each bandit may be not be fixed or uniform, subject instead to a possibly stochastic duration of activation before a change to a different bandit is allowed. The solution is based on generalized restart-in-state indices, and it utilizes a view of the problem not as "decisions over state space" but rather "decisions over time".

show abstract

Section: Introductionmentioning

confidence: 81%

Multi-Armed Bandits Under General Depreciation and Commitment

Cowan

Katehakis

2014

Prob. Eng. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…We mention the contributions of Katehakis and Veinott (the restart-in-state method, see Katehakis and Veinott 1987), Varaiya, Walrand and Buyukkoc (the-largest-remaining-index method, see Varaiya et al 1985), and Chen and Katehakis (the linear programming method, see Chen and Katehakis 1986). In this article we present the parametric linear programming method proposed in Kallenberg (1986).…”

Section: Theorem 22mentioning

confidence: 99%

Derman’s book as inspiration: some results on LP for MDPs

Kallenberg

2012

Ann Oper Res

View full text Add to dashboard Cite

In 1976 I was looking for a suitable subject for my PhD thesis. My thesis advisor Arie Hordijk and I found a lot of inspiration in Derman's book (Finite state Markovian decision processes, Academic Press, New York, 1970). Since that time I was interested in linear programming methods for Markov decision processes. In this article I will describe some results in this area on the following topics: (1) MDPs with the average reward criterion; (2) additional constraints; (3) applications. These topics are the main elements of Derman's book.

show abstract

“…The indices in (5), (6) are of Gittins type and are computable by a range of algorithms including the "restart-in-x" approach of Katehakis and Veinott (1987). When the state spaces Ω j , 1 ≤ j ≤ N , are finite the "largest-to-smallest" algorithm of Robinson (1982) (equivalently, the adaptive greedy algorithm of Bertsimas and Niño-Mora (1996)) is available.…”

Section: A General Model For a Single Conflict With Disengagementmentioning

confidence: 99%

“…The authors recommend an adapted version of the "restart-in-(n, δ)" approach to index computation proposed by Katehakis and Veinott (1987). See Glazebrook and Greatrix (1995) and refer to the first author for full details.…”

Section: Model 3 -'Shoot-look-shoot' For Redmentioning

confidence: 99%

See 1 more Smart Citation

Index Policies for Shooting Problems

Glazebrook

Kirkbride

Mitchell

et al. 2007

Operations Research

View full text Add to dashboard Cite

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, ABSTRACT (maximum 200 words)We consider a scenario in which a single Red wishes to shoot at a collection of Blue targets, one at a time, to maximise some measure of return obtained from Blues killed before Red's own (possible) demise. Such a situation arises in various military contexts such as the conduct of air defence by Red in the face of Blue SEAD (suppression of enemy air defences). A class of decision processes called multi-armed bandits has been previously deployed to develop optimal policies for Red in which she attaches a calibrating (Gittins) index to each Blue target and optimally shoots next at the Blue with largest index value. The current paper seeks to elucidate how a range of developments of index theory are able to accommodate features of such problems which are of practical military import. Such features include levels of risk to Red which are policy dependent, Red having imperfect information about the Blues she faces, an evolving population of Blue targets and the possibility of Red disengagement. The paper concludes with a numerical study which both compares the performance of (optimal) index policies to a range of competitors and also demonstrates the value to Red of (optimal) disengagement. NUMBER OF PAGES 2714. SUBJECT TERMS multi-armed bandits, Gitten Indices, suppression of enemy air defense PRICE CODE SECURITY CLASSIFICATION OF REPORT Unclassified SECURITY CLASSIFICATION OF THIS PAGE Unclassified SECURITY CLASSIFICATION OF ABSTRACT Unclassified LIMITATION OF ABSTRACT UL iiIndex policies for shooting problems AbstractWe consider a scenario in which a single Red wishes to shoot at a collection of Blue targets, one at a time, to maximise some measure of return obtained from Blues killed before Red's own (possible) demise. Such a situation arises in various military contexts such as the conduct of air defence by Red in the face of Blue SEAD (suppression of enemy air defences). A class of decision processes called multi-armed bandits has been previously deployed to develop optimal policies for Red in which she attaches a calibrating (Gittins) index to each Blue target and optimally shoots next at the Blue with largest index value. The current paper seeks to elucidate how a range of developments of index theory are able to accommodate features of such problems which are of practical military import. Such features include levels of risk to Red which are policy dependent, Red having imperfect information about the Blues she faces, an evolving population of Blue targets and the possibility of Red disengagement. The paper concludes with a numerical study which both c...

show abstract

The Multi-Armed Bandit Problem: Decomposition and Computation

Cited by 271 publications

References 5 publications

Multi-Armed Bandits Under General Depreciation and Commitment

Multi-Armed Bandits Under General Depreciation and Commitment

Derman’s book as inspiration: some results on LP for MDPs

Index Policies for Shooting Problems

Contact Info

Product

Resources

About