The Fragility of Optimized Bandit Algorithms

Abstract

Much of the literature on optimal design of bandit algorithms is based on minimization of expected regret. It is well known that designs that are optimal over certain exponential families can achieve expected regret that grows logarithmically in the number of arm plays, at a rate governed by the Lai-Robbins lower bound. In this paper, we show that when one uses such optimized designs, the associated algorithms necessarily have the undesirable feature that the tail of the regret distribution behaves like that of a truncated Cauchy distribution. Furthermore, for p > 1, the p'th moment of the regret distribution grows much faster than poly-logarithmically, in particular as a power of the number of sub-optimal arm plays. We show that optimized Thompson sampling and UCB bandit designs are also fragile, in the sense that when the problem is even slightly mis-specified, the regret can grow much faster than the conventional theory suggests. Our arguments are based on standard change-of-measure ideas, and indicate that the most likely way that regret becomes larger than expected is when the optimal arm returns below-average rewards in the first few arm plays that make that arm appear to be sub-optimal, thereby causing the algorithm to sample a truly sub-optimal arm much more than would be optimal.

Gaussian Imagination in Bandit Learning

“…A key assumption underlying our analysis is that the misspecified Gaussian distributions are sufficiently diffuse. The importance of diffuseness has also been highlighted in work on frequentist analysis of Thompson sampling and KL-UCB applied to bandits with independent arms (Honda and Takemura, 2013;Wager and Xu, 2021;Fan and Glynn, 2021). In contrast, our results apply to any agent and allow for generalization across arms.…”

A Simple and Optimal Policy Design with Safety against Heavy-tailed Risk for Multi-armed Bandits

“…We argue that instant-dependent consistency and light-tailed risk are incompatible. Built upon the analysis and results in Fan and Glynn (2021b), we find that a wide range of policies, including the standard UCB and SE policy, and the TS policy, suffer from heavy-tailed risk. More precisely, each of these three policies incurs a linear regret with probability Ω(poly(1/T )) = exp(−O(ln T )).…”

“…Our policy design resolves two issues that create a large regret: (i) spending too much time before correctly discarding a sub-optimal arm and (ii) wrongly discarding the optimal arm due to under-estimation (pointed out by Fan and Glynn (2021b)). Despite of the simplicity of our proposed policy design, the associated proof techniques are novel and may be useful for broader analysis on regret distribution and tail risk.…”

“…Our work is inspired by Fan and Glynn (2021b), who first provided a rigorous formulation to analyze heavy-tailed risk for bandit algorithms and showed that for any information-theoretically optimized bandit policy, the probability of incurring a linear regret is very heavy-tailed: at least Ω(1/T ).…”

“…Theorem 1 suggests that a consistent policy must have a risk tail heavier than an exponential one. To prove Theorem 1, we use a refined version of the change of measure argument originally introduced by Fan and Glynn (2021b). We consider two environments with θ = (1/2, 1) and θ =…”