Safe Linear Thompson Sampling with Side Information

Abstract

The design and performance analysis of bandit algorithms in the presence of stage-wise safety or reliability constraints has recently garnered significant interest. In this work, we consider the linear stochastic bandit problem under additional linear safety constraints that need to be satisfied at each round. We provide a new safe algorithm based on linear Thompson Sampling (TS) for this problem and show a frequentist regret of order O(d 3/2 log 1/2 d • T 1/2 log 3/2 T ), which remarkably matches the results provided by [Abeille et al., 2017] for the standard linear TS algorithm in the absence of safety constraints. We compare the performance of our algorithm with a UCB-based safe algorithm and highlight how the inherently randomized nature of TS leads to a superior performance in expanding the set of safe actions the algorithm has access to at each round.

Stochastic Bandits with Linear Constraints

“…Compared to the previous setting, our constraint is more relaxed (from high-probability to expectation), and as a result, it would be possible for us to obtain a solution with larger expected cumulative reward. We will have a detailed discussion on the relationship between these two settings and the similarities and differences of our results with those reported in Amani et al [2019] and Moradipari et al [2019] in Section 7.…”

“…In Figure 3, the reason that the cost evolution of OPB is the same as that of the optimal policy (middle) is that in this case, the cost of the best arm (arm 4) is equal to the constraint threshold τ = .2. As described in Section 1, our setting is the closest to the one studied by Amani et al [2019] and Moradipari et al [2019]. They study a slightly different setting, in which the mean cost of the action that the agent takes should satisfy the constraint, i.e., x t , µ * ≤ τ , not the mean cost of the policy it computes, i.e., x πt , µ * ≤ τ , as in our case.…”

“…Clearly, the setting studied in our paper is more relaxed, and thus, is expected to obtain more rewards. Moradipari et al [2019] propose a TS algorithm for their setting and prove an O(d 3/2 √ T /τ ) regret bound for it. They restrict themselves to linear bandits, i.e., A t = A, ∀t ∈ [T ], and the safe action being the origin, i.e., x 0 = 0 and c 0 = 0.…”

“…Here the constraint is stage-wise, and unlike the last two settings, is independent of the history. Amani et al [2019] and Moradipari et al [2019] have recently studied this setting for linear bandits and derived and analyzed explore-exploit [Amani et al, 2019] and Thompson sampling [Moradipari et al, 2019] algorithms for it.…”

Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs

“…One setting, referred to as conservative bandits [31,17,12], requires the cumulative reward to remain above a fixed percentage of the cumulative reward of a given baseline policy. Another setting is where each arm is associated with two unknown distributions (similar to our setting), generating reward and cost signals respectively [3,23,20,21].…”