Stochastic Bandits with Linear Constraints

Abstract

We study a constrained contextual linear bandit setting, where the goal of the agent is to produce a sequence of policies, whose expected cumulative reward over the course of T rounds is maximum, and each has an expected cost below a certain threshold τ . We propose an upper-confidence bound algorithm for this problem, called optimistic pessimistic linear bandit (OPLB), and prove an O( d √ T τ −c0 ) bound on its T -round regret, where the denominator is the difference between the constraint threshold and the cost of a known feasible action. We further specialize our results to multi-armed bandits and propose a computationally efficient algorithm for this setting. We prove a regret bound of O() for this algorithm in K-armed bandits, which is a √ K improvement over the regret bound we obtain by simply casting multi-armed bandits as an instance of contextual linear bandits and using the regret bound of OPLB. We also prove a lower-bound for the problem studied in the paper and provide simulations to validate our theoretical results.Preprint. Under review.

An Efficient Pessimistic-Optimistic Algorithm for Stochastic Linear Bandits with General Constraints

“…Since the computational complexity of the dual component depends on the number of constraints, but is independent of sizes of the contextual space, the action space, and the feature space, the overall computational complexity of our algorithm is similar to that of LinUCB in the unconstrained setting. Since our anytime cumulative constraint (2) is most similar to an anytime policy constraint in [31], we first compare our algorithm with OPLB proposed in [31]. OPLB needs to construct a safe policy set in each round.…”

“…it imposes a cumulative constraint in every round. This anytime cumulative constraint is most similar to an anytime policy constraint in [31] because the average cost of a policy is close to its mean after the policy has been applied for many rounds and the process converges, so can be viewed as a cumulative constraint on actions over many rounds (like ours). Furthermore, when our anytime cumulative constraint (2) is satisfied, our learner guarantees that the time-average cost is below a threshold in every round.…”

“…In general, the learner has more flexibility in the earlier rounds, close to that in the unconstrained setting. Another formulation is anytime constraints, which either require the expected cost of the action taken in each round is lower than a threshold [6] or the expected cost of the policy in each round is lower than a threshold [31]. We call them anytime action constraints and anytime policy constraints, respectively.…”

“…So the overall computational complexity of our algorithm is similar to the linear UCB for unconstrained stochastic linear bandits. * A preliminary version of this paper that considers the traditional multi-armed bandits can be found in [27].1 General constraints in this paper refer to that we do not assume the cost functions in the constraints are linear as in [6,31].…”

“…1 General constraints in this paper refer to that we do not assume the cost functions in the constraints are linear as in [6,31].…”