A Closer Look at the Worst-case Behavior of Multi-armed Bandit Algorithms

Abstract

One of the key drivers of complexity in the classical (stochastic) multi-armed bandit (MAB) problem is the difference between mean rewards in the top two arms, also known as the instance gap. The celebrated Upper Confidence Bound (UCB) policy is among the simplest optimism-based MAB algorithms that naturally adapts to this gap: for a horizon of play n, it achieves optimal O (log n) regret in instances with "large" gaps, and a near-optimal O √ n log n minimax regret when the gap can be arbitrarily "small." This paper provides new results on the arm-sampling behavior of UCB, leading to several important insights. Among these, it is shown that armsampling rates under UCB are asymptotically deterministic, regardless of the problem complexity. This discovery facilitates new sharp asymptotics and a novel alternative proof for the O √ n log n minimax regret of UCB. Furthermore, the paper also provides the first complete process-level characterization of the MAB problem under UCB in the conventional diffusion scaling. Among other things, the "small" gap worst-case lens adopted in this paper also reveals profound distinctions between the behavior of UCB and Thompson Sampling, such as an incomplete learning phenomenon characteristic of the latter.

The Countable-armed Bandit with Vanishing Arms

“…In contrast, the recently introduced countable-armed bandit (CAB) problem (Kalvit and Zeevi, 2020;de Heide et al, 2021) that our work is most closely related to, is fundamentally simpler owing to a finite set of arm-types; this is central to the achievability of logarithmic regret in the CAB problem. (Kalvit and Zeevi, 2020) study the CAB problem with K = 2 types, and propose an online adaptive algorithm that achieves an expected cumulative regret of O (log n) after any number of plays n. Notably, their algorithm does not require ex ante knowledge of (a lower bound on) the fraction α of "optimal" arms in the arm-reservoir, and its regret analysis relies purely on certain novel properties of the UCB algorithm (Auer et al, 2002a); these properties are elucidated in (Kalvit and Zeevi, 2021). The general CAB problem with multiple arm-types was subsequently studied in (de Heide et al, 2021), where it was shown that achieving logarithmic regret absent ex ante knowledge of α is impossible if K > 2.…”

The Fragility of Optimized Bandit Algorithms

“…In [27,10,14], diffusion approximations are given for TS and UCB algorithms under scaling regimes where the gaps between arm means shrink with the time horizon in a worst-case way. Although distributional characterizations of regret are obtained in these works (in terms of solutions to stochastic differential equations and random ordinary differential equations), the worst-case scaling regimes considered makes the results incomparable to those in this paper.…”