On Sequential Elimination Algorithms for Best-Arm Identification in Multi-Armed Bandits

Abstract

We consider the best-arm identification problem in multi-armed bandits, which focuses purely on exploration. A player is given a fixed budget to explore a finite set of arms, and the rewards of each arm are drawn independently from a fixed, unknown distribution. The player aims to identify the arm with the largest expected reward. We propose a general framework to unify sequential elimination algorithms, where the arms are dismissed iteratively until a unique arm is left. Our analysis reveals a novel performance measure expressed in terms of the sampling mechanism and number of eliminated arms at each round. Based on this result, we develop an algorithm that divides the budget according to a nonlinear function of remaining arms at each round. We provide theoretical guarantees for the algorithm, characterizing the suitable nonlinearity for different problem environments described by the number of competitive arms. Matching the theoretical results, our experiments show that the nonlinear algorithm outperforms the state-of-the-art. We finally study the side-observation model, where pulling an arm reveals the rewards of its related arms, and we establish improved theoretical guarantees in the pure-exploration setting.

Action-Manipulation Attacks on Stochastic Bandits

“…The algorithm is summarized in Algorithm 2. Compared with the original UCB algorithm in (3), the main difference is the additional term γ(μ(t − 1), N(t − 1)) in (15). We now highlight the main idea why our bandit algorithm works and the role of this additional term.…”

“…else 5: The user chooses arm I t to pull according (15). 6: end if 7: if The attacker decides to attack then 8: The attacker attacks and changes I t to I 0 t .…”

Exploratory Recommender Systems Based on Reinforcement Learning for Finding Research Topic

“…Xu [9] used MAB models to balance exploiting user data and protecting user privacy in dynamic pricing. Shahrampour [10] proposed a new algorithm for choosing the best arm of MAB, which outperforms the state-of-art. Lacerda [11] proposed an algorithm named as Multi-Objective Ranked Bandits for recommender systems.…”

A dynamic distributed energy storage control strategy for providing primary frequency regulation using multi‐armed bandits method

“…These studies seldom consider the uncertainty of users' behaviours, so this paper introduces an online learning method called multi-armed bandits (MAB) to solve the problem. MAB has shown effectiveness and merit in air conditioning demand aggregation [16] and many other sequential decisionmaking problems containing uncertain/unknown behavioural factors [17][18][19][20][21][22][23][24][25][26][27]. In reference [28], an adversarial MAB framework is applied to learn the signal response of thermal control loads for demand response in real-time.…”