Monotone Policies and Indexability for Bidirectional Restless Bandits

Abstract

Motivated by a wide range of applications, we consider a development of Whittle's restless bandit model in which project activation requires a state-dependent amount of a key resource, which is assumed to be available at a constant rate. As many projects may be activated at each decision epoch as resource availability allows. We seek a policy for project activation within resource constraints which minimises an aggregate cost rate for the system. Project indices derived from a Lagrangian relaxation of the original problem exist provided the structural requirement of indexability is met. Verification of this property and derivation of the related indices is greatly simplified when the solution of the Lagrangian relaxation has a state monotone structure for each constituent project. We demonstrate that this is indeed the case for a wide range of bidirectional projects in which the project state tends to move in a different direction when it is activated from that in which it moves when passive. This is natural in many application domains in which activation of a project ameliorates its condition, which otherwise tends to deteriorate or deplete. In some cases the state monotonicity required is related to the structure of state transitions, while in others it is also related to the nature of costs. Two numerical studies demonstrate the value of the ideas for the construction of policies for dynamic resource allocation, most especially in contexts which involve a large number of projects.

Indexability and Optimal Index Policies for a Class of Reinitialising Restless Bandits

“…These reinitializing bandits have some common features with models previously addressed: it is similar to the reward depletion and replenishment model presented in [4], and it also shares with bidirectional bandits in [5], the property that the active and passive actions produce opposite movements on the state space. Another related application is found in [7], where a new type of congestion control scheduling method based on a MARBP is proposed, motivated by the Internet flows behaving according to the Transmission Control Protocol, and thus admitting a reinitializing feature.…”

“…Denote by V β (φ 0 , λ, i) the expression (9) evaluated by setting t * (φ 0 , λ) = i. Notice that, solving problem (4), that is, finding the states that belong to A * (λ), is therefore equivalent to finding the maximum positive integer i such that it holds: (5) and (6), and given that V * β (φ 0 , λ) = V β (φ 0 , λ, i), using (4) we have that:…”

“…Furthermore, the indexability of special classes of MARBP has been specifically addressed and thoroughly studied using various approaches. These include some families of restless bandits which arise in machine maintenance and stochastic scheduling problems with switching costs, as those in Glazebrook, Ruiz-Hernandez, and Kirkbride [4], the bidirectional bandits introduced in Glazebrook, Hodge, and Kirkbride [5], the reinitializing bandits in Jacko and Sanso [7], and restless models in telecommunication and opportunistic spectrum access as in Liu and Zhao [10], among others. These papers are part of the body of literature that has contributed to a significant advance in the understanding of this property, yet as Liu, Weber, and Zhao [11] put it “[…] establishing indexability is still an open problem and often relies on numerical algorithms ”.…”

Restless bandits with controlled restarts: Indexability and computation of Whittle index

“…The condition appears to be a natural condition which should be satisfied by all models, but that is not the case [11]. Sufficient conditions for indexability have been investigated under specific modeling assumptions (two state fully or partially observed restless bandits [2], [6]; monotone bandits [2], [5], [13]; models with right-skeip free transitions [1], [14]; models with monotone or convex cost/reward [2], [13], [14], [16]- [18]; models satisfying partial conservation laws [19], [20]). Indexability for models arising in specific applications has been investigated in [1], [5], [14]- [18].…”

Capacity management for personalized services in education

“…In tackling this, a multi-armed bandit approach can be taken. For example, the bidirectional restless bandits (Glazebrook et al, 2013) could be a suitable fit for representing the learners' learning.…”