Minimax Optimal Algorithms for Adversarial Bandit Problem With Multiple Plays

Abstract

We investigate the adversarial bandit problem with multiple plays under semi-bandit feedback. We introduce a highly efficient algorithm that asymptotically achieves the performance of the best switching m-arm strategy with minimax optimal regret bounds. To construct our algorithm, we introduce a new expert advice algorithm for the multiple-play setting. By using our expert advice algorithm, we additionally improve the best-known high-probability bound for the multi-play setting by O( √ m). Our results are guaranteed to hold in an individual sequence manner since we have no statistical assumption on the bandit arm gains. Through an extensive set of experiments involving synthetic and real data, we demonstrate significant performance gains achieved by the proposed algorithm with respect to the state-of-the-art algorithms.Index Terms-Adversarial multi-armed bandit, multiple plays, switching bandit, minimax optimal, individual sequence manner. I. INTRODUCTION A. PreliminariesM ULTI-ARMED bandit problem is extensively investigated in the online learning [1]-[6] and signal processing [7]-[11] literatures, especially for the applications where feedback is limited, and exploration-exploitation must be balanced optimally. In the classical framework, the multi-armed bandit problem deals with choosing a single arm out of K arms at each round so as to maximize the total reward. We study the multiple-play version of this problem, where we choose an m sized subset of K arms at each round. We assume that r The size m is constant throughout the game and known a priori by the learner.r The order of arm selections does not have an effect on the arm gains.r The total gain of the selected m arms is the sum of the gains of the selected individual arms.

Generalized Translation and Scale Invariant Online Algorithm for Adversarial Multi-Armed Bandits

“…We study the multi-armed bandit problem in an online setting, where we sequentially operate on a stream of observations from an adversarial environment [54], i.e., we have no statistical assumptions on the loss sequence. To this end, we investigate the multi-armed bandit problem from a competitive algorithm perspective [13], [55]- [60].…”

Efficient Online-Bandit Strategies for Minimax Learning Problems

“…Several algorithms have been introduced to solve the CSB problem that arises from K = S n,k . This problem has been called the k-set problem [Combes et al, 2015], unordered slate [Kale et al, 2010] or bandits with multiple plays [Uchiya et al, 2010, Vural et al, 2019.…”

“…We use the EXP4.MP algorithm [Vural et al, 2019] for the p-player, which is a variation of EXP4 [Auer et al, 2002]. Each iteration of EXP4.MP has a computational cost of O(n log(n)) and a storage cost of O(n), with a high-probability regret bound of O( knT log(n/δ)).…”

“…This allows EXP4.MP to completely bypass handling a distribution over the exponentially large combinatorial set P k ([n]). DepRound was introduced in [Gandhi et al, 2006] and was used in the context of CSB by [Uchiya et al, 2010, Vural et al, 2019.…”

A dynamic distributed energy storage control strategy for providing primary frequency regulation using multi‐armed bandits method

“…These studies seldom consider the uncertainty of users' behaviours, so this paper introduces an online learning method called multi-armed bandits (MAB) to solve the problem. MAB has shown effectiveness and merit in air conditioning demand aggregation [16] and many other sequential decisionmaking problems containing uncertain/unknown behavioural factors [17][18][19][20][21][22][23][24][25][26][27]. In reference [28], an adversarial MAB framework is applied to learn the signal response of thermal control loads for demand response in real-time.…”