Asymptotically Optimal Multi-Armed Bandit Policies Under a Cost Constraint

Abstract

We develop asymptotically optimal policies for the multi armed bandit (MAB), problem, under a cost constraint. This model is applicable in situations where each sample (or activation) from a population (bandit) incurs a known bandit dependent cost. Successive samples from each population are iid random variables with unknown distribution. The objective is to design a feasible policy for deciding from which population to sample from, so as to maximize the expected sum of outcomes of n total samples or equivalently to minimize the regret due to lack on information on sample distributions, For this problem we consider the class of feasible uniformly fast (f-UF) convergent policies, that satisfy the cost constraint sample-path wise. We first establish a necessary asymptotic lower bound for the rate of increase of the regret function of f-UF policies. Then we construct a class of f-UF policies and provide conditions under which they are asymptotically optimal within the class of f-UF policies, achieving this asymptotic lower bound. At the end we provide the explicit form of such policies for the case in which the unknown distributions are Normal with unknown means and known variances. . Asymptotic optimality, finite horizon regret bounds, and a solution to an open problem. optimal Bayesian sequential change detection and identification rules. -armed bandit with budget constraint and variable costs. In AAAI-13, pages 232-238, 2013.Eugene A Feinberg, Pavlo O Kasyanov, and Michael Z Zgurovsky. Convergence of value iterations for total-cost mdps and pomdps with general state and action sets.

Optimal Data Driven Resource Allocation under Multi-Armed Bandit Observations

“…We develop a class of feasible policies that are shown to be asymptotically optimal within a large class of good policies that uniformly fast (UF) convergent, in the sense of Burnetas and Katehakis (1996) and Lai and Robbins (1985). The results in this paper extend the work in Burnetas et al (2017) which solved the case where there exists only one type of constraint for all bandits. Further, the class of block-UCB (b-UCB) feasible policies which are developed here and achieve the asymptotic lower bound in the regret have a simpler form and are easier to compute than those in Burnetas et al (2017).…”

“…The results in this paper extend the work in Burnetas et al (2017) which solved the case where there exists only one type of constraint for all bandits. Further, the class of block-UCB (b-UCB) feasible policies which are developed here and achieve the asymptotic lower bound in the regret have a simpler form and are easier to compute than those in Burnetas et al (2017). We also refer to Burnetas and Kanavetas (2012) where a consistent policy (i.e., with regret o(n)) for the case of a single linear constraint was constructed.…”

“…Following the approach in Burnetas et al (2017), we will establish in Theorem 5 below a lower bound M (θ) for the regret of any f-UF policy and construct a block UCB policy π 0 which is f-UF and its regret achieves this lower bound, i.e., lim n→∞ R π 0 (θ, n) log n ≤ M (θ), ∀θ.…”