Tsallis-INF: An Optimal Algorithm for Stochastic and Adversarial Bandits

Abstract

We derive an algorithm that achieves the optimal (up to constants) pseudo-regret in both adversarial and stochastic multi-armed bandits without prior knowledge of the regime and time horizon. The algorithm is based on online mirror descent with Tsallis entropy regularizer. We provide a complete characterization of such algorithms and show that Tsallis entropy with power α = 1/2 achieves the goal. In addition, the proposed algorithm enjoys improved regret guarantees in two intermediate regimes: the moderately contaminated stochastic regime defined by Seldin and Slivkins [22] and the stochastically constrained adversary studied by Wei and Luo [26]. The algorithm also achieves adversarial and stochastic optimality in the utility-based dueling bandit setting. We provide empirical evaluation of the algorithm demonstrating that it outperforms Ucb1 and Exp3 in stochastic environments. In certain adversarial regimes the algorithm significantly outperforms Ucb1 and Thompson Sampling, which exhibit almost linear regret.

Private Stochastic Convex Optimization: Efficient Algorithms for Non-smooth Objectives

“…We are now ready to show the main result of this section. Our proof follows the techniques of (Shamir and Zhang, 2013) combined with the analysis of FTRL in (Abernethy et al, 2015;Zimmert and Seldin, 2018). Let u ∈ W be a fixed vector in the convex set to be chosen later.…”

“…The proof of Theorem 5.1 follows the ideas for the analysis of FTRL in Abernethy et al ( 2015) and Zimmert and Seldin (2018), and combines them with the techniques used to obtain last iterate guarantees for stochastic gradient descent in Shamir and Zhang (2013). Since the analysis is somewhat standard we are going to devote the rest of the section to the privacy analysis.…”

Contextual Search in the Presence of Adversarial Corruptions

“…Lykouris et al [LMPL18] introduced a variant of the standard stochastic multi-armed bandit problem, where an adversary can corrupt a number of samples, and provided algorithms with learning rates that degrade according to the number of corruptions. The guarantees for stochastic multi-armed bandits were subsequently strengthened by Gupta et al [GKT19] and Zimmert and Seldin [ZS19], and the concept of adversarial corruptions has also been extended to several other settings including dynamic assortment optimization [CKW19], linear bandits [LLS19] and reinforcement learning [LSSS19]. Our work differs from these in that we use adversarial corruptions as a modeling tool to capture arbitrarily irrational agent behavior in game-theoretic settings.…”

Better Algorithms for Stochastic Bandits with Adversarial Corruptions

“…Bridging between the stochastic and adversarial settings in multi-armed bandits, and in online learning more generally, has been a topic of significant interest in recent years. Most research in this direction has focused on obtaining "the best of both worlds" guarantees (Bubeck and Slivkins, 2012;Seldin and Slivkins, 2014;Auer and Chiang, 2016;Seldin and Lugosi, 2017;Zimmert and Seldin, 2018;Zimmert et al, 2019). The goal there is to achieve the better of the bounds at the two extremes: the worst case O( √ T )-type bound on any problem instance, and the better O(log T )type bound whenever the instance is actually stochastic.…”

“…Adversarial contaminations similar to those considered here have been studied before in the context of bandit problems. Seldin and Slivkins (2014) and Zimmert and Seldin (2018) consider a "moderately contaminated" regime in which the adversarial corruptions do not reduce the gap min i =i ⋆ ∆ i by more than a constant factor at any point in time. This regime of contamination is very restrictive and, for example, precludes virtually any form of corruption in the early stages on learning.…”