Some indexable families of restless bandit problems

Abstract

In 1988 Whittle introduced an important but intractable class of restless bandit problems which generalise the multiarmed bandit problems of Gittins by allowing state evolution for passive projects. Whittle's account deployed a Lagrangian relaxation of the optimisation problem to develop an index heuristic. Despite a developing body of evidence (both theoretical and empirical) which underscores the strong performance of Whittle's index policy, a continuing challenge to implementation is the need to establish that the competing projects all pass an indexability test. In this paper we employ Gittins' index theory to establish the indexability of (inter alia) general families of restless bandits which arise in problems of machine maintenance and stochastic scheduling problems with switching penalties. We also give formulae for the resulting Whittle indices. Numerical investigations testify to the outstandingly strong performance of the index heuristics concerned.

Indexability of Restless Bandit Problems and Optimality of Whittle Index for Dynamic Multichannel Access

“…In the general context of RMBP, there is a rich literature on indexability. See [23] for the linear programming representation of conditions for indexability and [9] for examples of specific indexable restless bandit processes. Constant-factor approximation algorithms for RMBP have also been explored in the literature.…”

Prioritizing Hepatitis C Treatment in U.S. Prisons

“…We consider Whittle's index and other index-type policies. However, the implementation of Whittle's index is nontrivial because i) Whittle's index is only applicable to a restricted class of problems that satisfy the so-called indexability property, which is difficult to check in general (Whittle 1988, Glazebrook et al 2006; ii) the computation of Whittle's index involves repeated solution of single-bandit dynamic programs, which is difficult for our case since each single bandit has a multi-dimensional state space.…”

Index policies for discounted bandit problems with availability constraints

“…Because θ 1 (x, 1) = 1, Condition 2 is satisfied, and this version of Problem 2 is indexable. Glazebrook et al [11] formulated the same problem, slightly differently from us, as a restless bandit problem where one does not wait for the broken arm to be fixed before the reward stream is again available: if one plays a broken arm then he/she obtains its immediate reward minus the switching cost, and the arm is guaranteed to be available in the next period. However, the forms of their and our Whittle indices are the same.…”

“…We have developed the Whittle indices for Problems 1 and 2 in the previous sections. Here we discuss how to compute the indices in (9) and (11). For this purpose, we develop the restartin problem representation of the indices.…”

“…The restart-in problem representation of the Gittins index for the classical multiarmed bandit problem was introduced in [16]. The index in (11) is similar to the Gittins index; therefore, we first formulate it as a restart-in problem. We then propose a generalization of the restart-in problem representation for the Whittle index in (9).…”