Strategies for Safe Multi-Armed Bandits with Logarithmic Regret and Risk

Abstract

We investigate a natural but surprisingly unstudied approach to the multi-armed bandit problem under safety risk constraints. Each arm is associated with an unknown law on safety risks and rewards, and the learner's goal is to maximise reward whilst not playing unsafe arms, as determined by a given threshold on the mean risk.We formulate a pseudo-regret for this setting that enforces this safety constraint in a per-round way by softly penalising any violation, regardless of the gain in reward due to the same. This has practical relevance to scenarios such as clinical trials, where one must maintain safety for each round rather than in an aggregated sense.We describe doubly optimistic strategies for this scenario, which maintain optimistic indices for both safety risk and reward. We show that schema based on both frequentist and Bayesian indices satisfy tight gap-dependent logarithmic regret bounds, and further that these play unsafe arms only logarithmically many times in total. This theoretical analysis is complemented by simulation studies demonstrating the effectiveness of the proposed schema, and probing the domains in which their use is appropriate.

A Doubly Optimistic Strategy for Safe Linear Bandits

“…The bound above behaves inversely with respect to the three gaps, as well as with respect to ǫ. The following lower bound argues, via a reduction to the safe multi-armed bandit problem [CGS22], that the dependence on min(∆ I , Γ I ) in the above is tight for consistent algorithms ( §D.5).…”

“…Here, each constraint is associated with a notion of regret S i T = a i , x t − α i , and the overall regret is measured as max((max i S i T ), θ, x * − x t ). As noted by Pacchiano et al [PGBJ21] and Chen et al [CGS22], the main disadvantage of this formulation from our perspective arises from the fact that constraint violations are aggregated. This functionally means that it is okay for effective algorithms to alternate between actions that have large reward & poor safety, and actions that have poor reward but good safety.…”

“…This functionally means that it is okay for effective algorithms to alternate between actions that have large reward & poor safety, and actions that have poor reward but good safety. Indeed, this behaviour is exhibited quite explicitly in prior work [CGS22]. Such behaviour intrinsically does not model scenarios such as clinical trials or engineering design, since in these settings the interest is in maintaining the safety for most instances.…”

“…While we note that doubly optimistic approaches have previously appeared in the context of aggregate constraints [e.g. AD14] and for safe multi-armed bandits [CGS22], ours is the first work employing a doubly-optimistic approach in per-round safety constrained linear bandits 1 .…”

“…Prior works in this context can be grouped into two categories-those that deal with aggregated constraints and other more recent works that account for per-round constraint. As noted in recent work [PGBJ21;CGS22] the per-round constraint requirement is stringent, and prior strategies that utilize aggregate constraints for this problem [BKS13; BLS14; AD14; ADL16; AD16] are not compatible. On the other hand, prior works accounting for per-round safety exclusively rely upon optimistic-pessimistic strategies to ensure that these per-round constraints are met with high probability for each round [AAT19; MAAT21; PGBJ21].…”