Index policies for discounted bandit problems with availability constraints

Abstract

Multi-armed bandit problem is studied when the arms are not always available. The arms are first assumed to be intermittently available with some state/action-dependent probabilities. It is proven that no index policy can attain the maximum expected total discounted reward in every instance of that problem. The Whittle index policy is derived, and its properties are studied.Then it is assumed that arms may break down, but repair is an option at some cost, and the new Whittle index policy is derived. Both problems are indexable. The proposed index policies cannot be dominated by any other index policy over all multi-armed bandit problems considered here. Whittle indices are evaluated for Bernoulli arms with unknown success probabilities.

Multi-armed Bandits with Constrained Arms and Hidden States

“…We next show the indexability and compute the closed form expression for the index of a single-armed rested bandit. The proof of index computation is along the lines of [13].…”

“…For example, in a machine-repair problem one may not able to schedule a task on some of the machines due to machine breakdown. Such consideration has been made in [13]. In queuing systems, the controller may not be able to schedule jobs to some servers due to server breakdown, [14], [15].…”

“…In constrained bandits [13], [14], [15], each state is defined as a pair (X(t), Y (t)), where X(t) represents the state of arm and Y (t) represents availability of an arm at time t. Time is discretized in [13] while it is continuous in the models of [14], [15]. The state (X(t), Y (t)) is assumed to be observable.…”

“…The state (X(t), Y (t)) is assumed to be observable. Under some assumptions on model parameters the index policy is analyzed in [13], [14], [15]. In this paper we consider a hidden Markov model, where state X(t) of the arm is not observable but the availability of the arm is observable.…”

Optimal Selection of Obsolescence Mitigation Strategies Using a Class of Bandit Models

“…However, at any stage, the decision maker chooses only one ann, and there is no passive reward for the anns which are not chosen. Dayanik et al (2007) have proved that when the passive rewards are equal to zero, the Whittle's index converges to Gitlin's index. In fact, for the optimal selection of obsolescence strategies, since only one strategy can be chosen at any period and passive anns carry no reward; there is no difference between Whittle's and Gitlin's indices.…”