On Kernelized Multi-Armed Bandits with Constraints

Abstract

We study a stochastic bandit problem with a general unknown reward function and a general unknown constraint function. Both functions can be non-linear (even non-convex) and are assumed to lie in a reproducing kernel Hilbert space (RKHS) with a bounded norm. This kernelized bandit setup strictly generalizes standard multi-armed bandits and linear bandits. In contrast to safety-type hard constraints studied in prior works, we consider soft constraints that may be violated in any round as long as the cumulative violations are small, which is motivated by various practical applications. Our ultimate goal is to study how to utilize the nature of soft constraints to attain a finer complexity-regret-constraint trade-off in the kernelized bandit setting. To this end, leveraging primal-dual optimization, we propose a general framework for both algorithm design and performance analysis. This framework builds upon a novel sufficient condition, which not only is satisfied under general exploration strategies, including upper confidence bound (UCB), Thompson sampling (TS), and new ones based on random exploration, but also enables a unified analysis for showing both sublinear regret and sublinear or even zero constraint violation. We demonstrate the superior performance of our proposed algorithms via numerical experiments based on both synthetic and real-world datasets. Along the way, we also make the first detailed comparison between two popular methods for analyzing constrained bandits and Markov decision processes (MDPs) by discussing the key difference and some subtleties in the analysis, which could be of independent interest to the communities.

Interference Constrained Beam Alignment for Time-Varying Channels via Kernelized Bandits

“…Our dynamic regret bounds match the order of the regret in the nonstationary Gaussian Process bandits without any constraints [17], [18], [21]. It also generalizes the kernelized bandits with constraints [19] in stationary environments to the time-varying case.…”

“…We compare the performance of the proposed algorithm with GP-UCB algorithm with Constraints (i.e. CKB-UCB algorithm in [19]). We perform the comparison under both abruptly-changing environments and the slowly-varying scenarios.…”

“…Similarly, to tackle non-stationarity, there are mainly three methods: restart [17], sliding window [17] and weight penalty [18]. This problem becomes more challenging in practical applications where there are unknown soft constraints [19]. In this paper, we generalize previous non-stationary results to the case with a general unknown time-varying constraint function.…”

“…In the online and non-stationary setting, it is difficult to directly solve the problem defined in (2) by choosing the best policy at time t. This is because both the reward and constraint function are time-varying and the noisy observation is revealed only after a policy is executed. In addition, since the constraint function g t is unknown, the constraint g t (x t ) ≤ 0 may not be satisfied in every time t. Instead, we aim to satisfy the long-term constraint T t=1 g t (x t ) ≤ 0 over a given period T [19]. We define the dynamic regret R(T ) as the difference of total reward between our policy {x t } T t=1 and the best policy {π * t } T t=1 .…”

“…Our main results are obtained via novel technical contributions. In particular, a direct combination of techniques from constrained bandits [19] and non-stationary bandits [17] would fail. This is because the standard 'time-average' trick [22] does not hold when the unknown functions are time-varying.…”