Deterministic Sequencing of Exploration and Exploitation for Multi-Armed Bandit Problems

Abstract

In the Multi-Armed Bandit (MAB) problem, there is a given set of arms with unknown reward models. At each time, a player selects one arm to play, aiming to maximize the total expected reward over a horizon of length T . An approach based on a Deterministic Sequencing of Exploration and Exploitation (DSEE) is developed for constructing sequential arm selection policies. It is shown that for all light-tailed reward distributions, DSEE achieves the optimal logarithmic order of the regret, where regret is defined as the total expected reward loss against the ideal case with known reward models. For heavy-tailed reward distributions, DSEE achieves O(T 1/p ) regret when the moments of the reward distributions exist up to the pth order for 1 < p ≤ 2 and O(T 1/(1+p/2) ) for p > 2. With the knowledge of an upperbound on a finite moment of the heavy-tailed reward distributions, DSEE offers the optimal logarithmic regret order. The proposed DSEE approach complements existing work on MAB by providing corresponding results for general reward distributions. Furthermore, with a clearly defined tunable parameter-the cardinality of the exploration sequence, the DSEE approach is easily extendable to variations of MAB, including MAB with various objectives, decentralized MAB with multiple players and incomplete reward observations under collisions, MAB with unknown Markov dynamics, and combinatorial MAB with dependent arms that often arise in network optimization problems such as the shortest path, the minimum spanning, and the dominating set problems under unknown random weights. Index TermsMulti-armed bandit, regret, deterministic sequencing of exploration and exploitation, decentralized multi-armed bandit, restless multi-armed bandit, combinatorial multi-armed bandit.

Mean-variance and value at risk in multi-armed bandit problems

“…However, the difference between the pseudo-regret defined in [11] and regret in its original definition is in the order of O( √ T ). They have also shown that a variation of the DSEE policy developed in [15] for risk-neutral MAB achieves O(T 2/3 ) regret performance without the positive difference assumption. While [11] only focuses on policy development, this paper also provides tight lower bounds on both the asymptotic and finite-time regret performance which serve as fundamental limits for gauging the optimality of learning policies.…”

“…A variation of the DSEE policy developed in [15] for risk-neutral MAB was considered in [11] and was shown to achieve O(T 2/3 ) finite-time regret performance. In the MV-DSEE policy, time is divided into two interleaving sequences: an exploration sequence denoted by E(t) and an exploitation sequence.…”

A Dynamic Spectrum Access Optimization Model for Cognitive Radio Wireless Sensor Network

“…In [18] the authors presented multiuser based spectrum access model. When multi users are presented in network it induces Collison.…”

“…When multi users are presented in network it induces Collison. To address this in [18] the presented an adaptive random access model and in [19] authors presented fair access model and [20] authors presented priority access model to reduce the collision among cognitive users. It is seen from exiting research [21] [22] that most of these schemes are limited to provide only one channel at a time to a cognitive user.…”

Online learning for network optimization under unknown models

“…For learning the shortest path, we can simply treat each path as an arm and directly apply existing MAB policies developed in [2][3][4][5][6]. This approach, however, results in a regret growing linearly with the number of paths, thus exponentially with the number of edges in the worst case.…”