Restless bandits: activity allocation in a changing world

Abstract

We consider a population of n projects which in general continue to evolve whether in operation or not (although by different rules). It is desired to choose the projects in operation at each instant of time so as to maximise the expected rate of reward, under a constraint upon the expected number of projects in operation. The Lagrange multiplier associated with this constraint defines an index which reduces to the Gittins index when projects not being operated are static. If one is constrained to operate m projects exactly then arguments are advanced to support the conjecture that, for m and n large in constant ratio, the policy of operating the m projects of largest current index is nearly optimal. The index is evaluated for some particular projects.

Dynamic Assortment with Demand Learning for Seasonal Consumer Goods

“…That is, in each period the constraint S s=1 u s ≤ N is replaced by Ɛ S s=1 u s ≤ N , where the expectation is with respect to all possible states weighted by the probability of reaching each one of them under a given (primal) policy. This fact has been observed by several authors in various other settings (e.g., Whittle 1988, Castañon 1997, and a proof of this equivalence for the finite horizon multiarmed bandit is available in Caro (2005). We also point out that Adelman and Mersereau (2004) provide an alternative linear programming-based bound that is shown to be tighter (or not worse) than (7), but requires more extensive computations.…”

“…The underlying concepts involved are similar to those of the well-established theory of duality for general nonlinear optimization problems (see Bertsekas 1999). The approach dates back to at least the late 1980s with the independent work done by Karmarkar (1987) on a finite horizon multilocation inventory problem and the seminal paper of Whittle (1988) on restless bandits. For more accounts of successful applications of this methodology, see Castañon (1997), Bertsimas and Mersereau (2004), and the references therein.…”

“…That is, at any time t, include in the assortment the N products with the largest desirability indices t s . Such type of heuristic policy was first introduced by Whittle (1988) for an infinite horizon model with restless bandits. The specific index he proposes is also derived as a break-even Lagrange multiplier, and its exact computation is a complicated task that can only be done numerically, as is the case in our model.…”

“…The main reason for this is that it is not clear at all how other policies can be adapted to our specific environment and furthermore extended to include lead times, switching costs, or substitution effects. These considerations led us to exclude in particular the policies described in Anantharam et al (1987) and Whittle (1988). One could also think of policies based on the dual upper bound (7).…”

“…However, they assume a Beta-Bernoulli learning model, whereas we use the Gamma-Poisson model, and they do not provide a suboptimality bound for their derived policy. Finally, although Whittle (1988) focuses on restless bandits, we owe much to his work in that he also uses a Lagrangian relaxation and considers multiple plays per stage.…”