Lipschitz Bandits without the Lipschitz Constant

Abstract

Abstract. We consider the setting of stochastic bandit problems with a continuum of arms indexed by [0, 1] d . We first point out that the strategies considered so far in the literature only provided theoretical guarantees of the form: given some tuning parameters, the regret is small with respect to a class of environments that depends on these parameters. This is however not the right perspective, as it is the strategy that should adapt to the specific bandit environment at hand, and not the other way round. Put differently, an adaptation issue is raised. We solve it for the special case of environments whose mean-payoff functions are globally Lipschitz. More precisely, we show that the minimax optimal orders of magnitude L d/(d+2) T (d+1)/(d+2) of the regret bound over T time instances against an environment whose mean-payoff function f is Lipschitz with constant L can be achieved without knowing L or T in advance. This is in contrast to all previously known strategies, which require to some extent the knowledge of L to achieve this performance guarantee.

On Two Continuum Armed Bandit Problems in High Dimensions

“…In terms of n, it nearly matches 2 the (n 1+k 2+k ) lower bound [13], for k-variate Lipschitz continuous mean reward functions. As explained earlier, the per-round regret R(n)/n approaches zero (as n increases), at a rate exponential in k. Thus for k d, we avoid the curse of dimensionality.…”

“…Say the linear parameter matrix A, or even the sub-space spanned by its rows, was known. We then know a lower bound of (n 1+k 2+k ) on regret, for k-variate Lipschitz continuous mean rewards [13]. In terms of n, our bound nearly matches this lower bound, albeit for a slightly restricted class of Lipschitz continuous mean reward functions.…”

“…This bound holds for both the stochastic and adversarial models and is nearly optimal. To see this, we note that Bubeck et al [13] showed a precise lower bound of (n d+1 d+2 ) after n = (2 d ) plays for stochastic continuum armed bandits, with d-variate Lipschitz continuous mean reward functions, defined over [0, 1] d . In our setting though, the reward functions depend on an unknown subset of k coordinate variables hence any algorithm after n = (2 k ) plays would incur worst case regret of (n k+1 k+2 ).…”

“…To see this, let S = [−1, 1] d and consider a time invariant reward function that is zero in all but one orthant O of S. More precisely, let R(n) denote the cumulative regret incurred by the algorithm after n rounds. Bubeck et al [13] showed that R(n) = (n d+2 ) which means that it converges to zero at a rate at least exponentially slow in d. This curse of dimensionality is avoided by reward functions possessing more structure, two popular cases being linear reward functions (see for example [2,33]) and convex reward functions (see for example [21,26]) for which the regret is polynomial in d and sub-linear in n.…”

Global Continuous Optimization with Error Bound and Fast Convergence

“…As noted by Hansen and Jaumard (1995), it is unclear if this approach provides any advantage, considering that other successful heuristics are already available. This argument still applies to this day to relatively recent algorithms such as those by Kvasov et al (2003) and Bubeck, Stoltz, and Yu (2011).…”