Multi-user lax communications: A multi-armed bandit approach

Abstract

Inspired by cognitive radio networks, we consider a setting where multiple users share several channels modeled as a multi-user multi-armed bandit (MAB) problem. The characteristics of each channel are unknown and are different for each user. Each user can choose between the channels, but her success depends on the particular channel chosen as well as on the selections of other users: if two users select the same channel their messages collide and none of them manages to send any data. Our setting is fully distributed, so there is no central control. As in many communication systems, the users cannot set up a direct communication protocol, so information exchange must be limited to a minimum. We develop an algorithm for learning a stable configuration for the multi-user MAB problem. We further offer both convergence guarantees and experiments inspired by real communication networks, including comparison to state-of-the-art algorithms. arXiv:1504.08167v2 [cs.LG] 2 Dec 2015 2 Model and formulationWe now describe the model, the assumptions accompanying it and our goal. System and usersWe model a communication network with K channels, servicing N independent users. Our work is based on the assumption that K ≥ N , which is reasonable since without it, implementing a time division based mechanism is necessary. Once such a mechanism is applied, the assumption that K ≥ N is valid again. Time is slotted and users' clocks are synchronized, also a mild assumption for modern communication systems.The communication network consists of K channels, where only one user can transmit over a certain channel during a single time slot. Each transmission yields a reward, which we assume to be stochastic.The users are a group of N independent, selfish agents. Their observations are local, consisting only of the history of their actions and rewards. In addition, they do not know the number of users they share a network with. There is no central control managing their use of the network, and they do not have direct communication with each other.A key characteristic of our model is that the expected reward a channel yields depends not only on the identity of the channel, but also on the identity of the user. Formally, the rewards of the channels are Bernoulli random variables with expected values {µ n,k }, where n ∈ {1, . . . , N } and k ∈ {1, . . . , K}. This property reflects the fact that in real-life users may experience location-based disturbances, manifested in different reward distributions for the same channel.We model the users' sharing resources through the representation of the communication network by a single bandit. This means that two users attempting to access the same channel at the same time, will experience a collision. In our model, the result of a collision is complete loss of communication for that time slot for the colliding users, i.e., zero reward. A user n that accesses a channel k alone during a certain time slot will receive a reward drawn i.i.d. from a Bernoulli distribution with expected value µ n,k . Through...

Multi-Player Multi-Armed Bandits for Stable Allocation in Heterogeneous Ad-Hoc Networks

“…However the main drawback of dE3, M-ETC, ESER is that they require frequent exchange of information between the users resulting in significant throughput loss due to signal overheads. Coordinated Stable Marriage MAB (CSM_MAB) algorithm in [5] overcomes the need for frequent communications by aiming to achieve stable allocation rather than optimal allocation. However, it requires that all users to simultaneously sense all channels (wideband sensing) which is, as discussed later in this section, computationally complex and expensive and hence may not be suitable for low cost, battery operated user terminals.…”

“…However, it requires that all users to simultaneously sense all channels (wideband sensing) which is, as discussed later in this section, computationally complex and expensive and hence may not be suitable for low cost, battery operated user terminals. Our goal in this work is to achieve stable allocation as in [5] in a heterogeneous network with minimum signaling overhead and using simpler narrowband radios that are well suited for ad-hoc networks with power constraints.…”

“…Our contributions in this work is to design distributed algorithms for heterogeneous channels for both static and dynamic network scenario. Furthermore, in our work each user can sense and transmit over only one channel in each time slot (narrowband sensing) which is more realistic and computationally efficient than wideband sensing in [5]. Our novel block structure makes the protocol simple, easy to implement and achieves performance better than that achieved by CSM_MAB using wideband sensing [5].…”

“…• We validate the performance of both dSOC_SN and dSOC_DN algorithms through extensive simulations. They outperform the state-of-the-art algorithm in [5] for heterogeneous ad-hoc networks. • Our algorithms need low complexity single antenna narrowband radio compared to wideband radios in existing works such as [5].…”

“…They outperform the state-of-the-art algorithm in [5] for heterogeneous ad-hoc networks. • Our algorithms need low complexity single antenna narrowband radio compared to wideband radios in existing works such as [5]. Unlike [6][7][8], the dSOC_DN algorithm does not require global clock synchronization for new users thereby further reducing the algorithm and radio complexity.…”