Stochastic scheduling: A short history of index policies and new approaches to index generation for dynamic resource allocation

Glazebrook, K. D.; Hodge, D. J.; Kirkbride, C.; Minty, R. J.

doi:10.1007/s10951-013-0325-1

Cited by 17 publications

(18 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Whittle index is a generalization of the Gittins index to the restless bandit case in which state processes (or belief states) evolve irrespective of whether an arm is being played or not -as opposed to the classical multiarmed bandit problem [6,7] in which the states of unplayed arms do not evolve. In order to devise a heuristic policy for a restless multiarmed bandit problem, Whittle [16] considered arms separately (i.e., he considered d decoupled one-armed bandits), and introduced a subsidy paid for leaving the arm under consideration passive.…”

Section: Whittle Indexmentioning

confidence: 99%

Exploration vs Exploitation with Partially Observable Gaussian Autoregressive Arms

Kühn¹,

Mandjes²,

Nazarathy³

2015

Proceedings of the 8th International Conference on Performance Evaluation Methodologies and Tools

View full text Add to dashboard Cite

We consider a restless bandit problem with Gaussian autoregressive arms, where the state of an arm is only observed when it is played and the state-dependent reward is collected. Since arms are only partially observable, a good decision policy needs to account for the fact that information about the state of an arm becomes more and more obsolete while the arm is not being played. Thus, the decision maker faces a tradeoff between exploiting those arms that are believed to be currently the most rewarding (i.e. those with the largest conditional mean), and exploring arms with a high conditional variance. Moreover, one would like the decision policy to remain tractable despite the infinite state space and also in systems with many arms. A policy that gives some priority to exploration is the Whittle index policy, for which we establish structural properties. These motivate a parametric index policy that is computationally much simpler than the Whittle index but can still outperform the myopic policy. Furthermore, we examine the many-arm behavior of the system under the parametric policy, identifying equations describing its asymptotic dynamics. Based on these insights we provide a simple heuristic algorithm to evaluate the performance of index policies; the latter is used to optimize the parametric index.

show abstract

Section: Whittle Indexmentioning

confidence: 99%

Exploration vs Exploitation with Partially Observable Gaussian Autoregressive Arms

Kühn¹,

Mandjes²,

Nazarathy³

2015

Proceedings of the 8th International Conference on Performance Evaluation Methodologies and Tools

View full text Add to dashboard Cite

show abstract

“…The recent survey paper [21] is a good up-to-date reference on the application of index policies in scheduling.…”

Section: Related Workmentioning

confidence: 99%

“…Set θ k = θ k for all k and consider the index multiplied by θ k as θ k → 0. A heuristic is then to give priority according to the index as given in (21).…”

Section: M/m/1 Multi-class Queuementioning

confidence: 99%

“…In case of linear holding costs C k (n k ) = c k n k , the index (21) coincides with the c k µ k -rule. For general holding cost the index in (21) was also obtained in Glazebrook et.al [1] (see also Section [19, Section 6.5]) by carrying out a model-dependent analysis, which consists in considering first the total discounted holding cost criterion, calculating the corresponding Whittle's index, and afterwards taking the limit in the discounting factor.…”

Section: M/m/1 Multi-class Queuementioning

confidence: 99%

“…Numerical example. In Table 1 we compare the suboptimality of the C (n)µ-rule and index-rule (21) in an M/M/1 queue without abandonments. Note that when θ k = 0, for all k, we need to assume K k=1 ρ k < 1 in order to assure stability of the system.…”

Section: M/m/1 Multi-class Queuementioning

confidence: 99%

See 2 more Smart Citations

Index policies for a multi-class queue with convex holding cost and abandonments

Larrañaga

Ayesta²,

Verloop

2014

The 2014 ACM International Conference on Measurement and Modeling of Computer Systems

View full text Add to dashboard Cite

We investigate a resource allocation problem in a multi-class server with convex holding costs and user impatience under the average cost criterion. In general, the optimal policy has a complex dependency on all the input parameters and state information. Our main contribution is to derive index policies that can serve as heuristics and are shown to give good performance. Our index policy attributes to each class an index, which depends on the number of customers currently present in that class. The index values are obtained by solving a relaxed version of the optimal stochastic control problem and combining results from restless multiarmed bandits and queueing theory. They can be expressed as a function of the steady-state distribution probabilities of a one-dimensional birth-and-death process. For linear holding cost, the index can be calculated in closed-form and turns out to be independent of the arrival rates and the number of customers present. In the case of no abandonments and linear holding cost, our index coincides with the cµ-rule, which is known to be optimal in this simple setting. For general convex holding cost we derive properties of the index value in limiting regimes: we consider the behavior of the index (i) as the number of customers in a class grows large, which allows us to derive the asymptotic structure of the index policies, and (ii) as the abandonment rate vanishes, which allows us to retrieve an index policy proposed for the multiclass M/M/1 queue with convex holding cost and no abandonments. In fact, in a multi-server environment it follows from recent advances that the index policy is asymptotically optimal for linear holding cost. To obtain further insights into the index policy, we consider the fluid version of the relaxed problem and derive a closed-form expression for the fluid index. The latter coincides with the stochastic model * The PhD fellowship of Maialen Larrañaga is funded by a research grant of the Foundation Airbus Group in case of linear holding costs. For arbitrary convex holding cost the fluid index can be seen as the Gcµ/θ-rule, that is, including abandonments into the generalized cµ-rule (Gcµ-rule). Numerical experiments show that our index policies become optimal as the load in the system increases.

show abstract

Optimal activation of halting multi‐armed bandit models

Cowan,

Katehakis,

Ross

2023

Naval Research Logistics

View full text Add to dashboard Cite

show abstract

Stochastic scheduling: A short history of index policies and new approaches to index generation for dynamic resource allocation

Cited by 17 publications

References 32 publications

Exploration vs Exploitation with Partially Observable Gaussian Autoregressive Arms

Exploration vs Exploitation with Partially Observable Gaussian Autoregressive Arms

Index policies for a multi-class queue with convex holding cost and abandonments

Optimal activation of halting multi‐armed bandit models

Contact Info

Product

Resources

About