The Budgeted Multi-armed Bandit Problem

Abstract

The following coins problem is a version of a multi-armed bandit problem where one has to select from among a set of objects, say classifiers, after an experimentation phase that is constrained by a time or cost budget. The question is how to spend the budget. The problem involves pure exploration only, differentiating it from typical multi-armed bandit problems involving an exploration/exploitation tradeoff [BF85]. It is an abstraction of the following scenarios: choosing from among a set of alternative treatments after a fixed number of clinical trials, determining the best parameter settings for a program given a deadline that only allows a fixed number of runs; or choosing a life partner in the bachelor/bachelorette TV show where time is limited. We are interested in the computational complexity of the coins problem and/or efficient algorithms with approximation guarantees. The Coins ProblemWe are given:-A collection of n independent coins, indexed by the set I, where each coin is specified by a probability density function (prior) over its head probability. The priors of the different coins are independent, and they can be different for different coins. -A budget b on the total number of coin flips.We assume the tail and the head outcomes correspond to receiving no reward and a fixed reward (1 unit) respectively. We are allowed a trial/learning period, constrained by the budget, for the sole purpose of experimenting with the coins, i.e., we do not collect rewards in this period. At the end of the period, we are allowed to pick only a single coin for all our future flips (reward collection).Let the actual head probability of coin i be θ i . We define the regret from picking coin i to be θ * −θ i , where θ * = max j∈I θ j . As we have the densities only, we basically seek to make coin flip decisions and a final choice that lead to minimizing our expected regret. It is easy to verify that when the budget is 0, the choice of coin that minimizes expected regret is one with maximum expected head probability over all the coins, i.e., max i E(Θ i ), where Θ i denotes the random variable corresponding to head probability of coin i, and the expectation E(Θ i ) is taken over the density for coin i.A strategy is a prescription of which coin to flip given all the coins' flip outcomes so far. A strategy may be viewed as a finite directed rooted tree, where each node indicates

Greedy Confidence Pursuit: A Pragmatic Approach to Multi-bandit Optimization

“…Figure 3 supports the predictiveness of (12) with respect to relative empirical costs. Note that, if one assumes the same target confidence τ for all bandits b i ∈ B, then accounting for τ inĉ i would not affect the ordering of bandits according tô c i , as an "easier" bandit according to (12) would also have a smaller expected completion cost for any value of τ . By using (12), we also ignore the effort previously expended on a given bandit.…”

“…While considering the number of trials already spent on a bandit could improve on the performance of (12), it would require steps to avoid the "sunk-cost" fallacy of economics, as manifested by premature commitment to bandits wrongly identified as "easy". (12). From darkest to lightest the points represent selecting the top 1, 3, and 5 arms of a 10-armed bandit.…”

“…(3). Note that (12) captures dependence on the subset size m through its use of Γ i and thatĉ i becomes stochastic when sampled with respect to the perbandit posteriors over returns and gaps. A theoretical analysis of our allocation process is beyond the scope of this paper, but the properties of posterior sampling described in Section 3 suggest it will efficiently direct trials towards the bandits with minimalĉ i .…”

“…We begin our empirical examination of greedy confidence pursuit with tests supporting (12) as an approximate completion cost. These tests were based on selecting the top 1, 3, and 5 arms of 10-armed bandits, with returns distributed as in Section 4.…”

“…The machine learning community has developed ways to formulate and address variations on this problem. For example, budgeted learning [12] and subsequent work considers the problem of active learning when a fixed budget is given for probing which model among a collection of models is best for a given task.…”