Distributed Learning over Markovian Fading Channels for Stable Spectrum Access

Abstract

We consider the problem of multi-user spectrum access in wireless networks. The bandwidth is divided into K orthogonal channels, and M users aim to access the spectrum. Each user chooses a single channel for transmission at each time slot. The state of each channel is modeled by a restless unknown Markovian process. Previous studies have analyzed a special case of this setting, in which each channel yields the same expected rate for all users. By contrast, we consider a more general and practical model, where each channel yields a different expected rate for each user. This model adds a significant challenge of how to efficiently learn a channel allocation in a distributed manner to yield a global system-wide objective. We adopt the stable matching utility as the system objective, which is known to yield strong performance in multichannel wireless networks, and develop a novel Distributed Stable Strategy Learning (DSSL) algorithm to achieve the objective. We prove theoretically that DSSL converges to the stable matching allocation, and the regret, defined as the loss in total rate with respect to the stable matching solution, has a logarithmic order with time. Finally, simulation results demonstrate the strong performance of the DSSL algorithm. I. INTRODUCTIONWe consider the spectrum access problem, where a shared bandwidth is divided into K orthogonal channels (i.e., subbands), and M users want to access the spectrum, where K ≥ M . Each channel is modeled by a Finite-State Markovian Channel (FSMC), which is independent and non-identically distributed across channels. The FSMC is a tractable model widely used to capture the time-varying behavior of a radio communication channel [2], [3]. It is often employed to model radio channel dynamics due to primary user occupancy effects in hierarchical cognitive radio networks (where the M secondary (unlicensed) users are cognitive in terms of learning and adapting good access strategies), or the external interference effects in the open sharing model among M users in the wireless network (e.g., ISM band) [4], [5]. At each time step, each user experiences a different transmission rate over each channel depending on its FSMC distribution, where the FSMC parameters (i.e., the transition probabilities that govern the Markov chain) are unknown. At each time step, each user is allowed to choose one channel to access, and observe the instantaneous channel state. If two users or more access the same channel at the same time, a collision occurs and the achievable rate is zero.We adopt the stable matching utility (see Section II for details) as the system objective, which is known to yield strong Tomer Gafni and Kobi Cohen are with the School of Electrical and Computer Engineering,

Restless Multi-Armed Bandits under Exogenous Global Markov Process

“…The extended RMAB model considered here is a generalization of the classic MAB problem [9]- [13]. RMAB problems have been studied under both the non-Bayesian [5]- [7], [14]- [18], and Bayesian [19]- [21], [22]- [28] settings. Under the non-Bayesian setting, special cases of Markovian dynamics have been studied in [6], [14], [16].…”

Federated Learning: A Signal Processing Perspective

“…For instance, intriguing research questions are: (i) How to develop dynamic spectrum access that integrates over-theair and orthogonal transmissions when coherent over-the-air transmission over the entire network is not efficient; and (ii) How to develop spatio-temporal dynamic spectrum access strategies when users experience deep fading in time and frequency. For example, a promising direction is to model the fading channel as a known or unknown Markovian process, for which theoretical performance measures for channel allocation strategies can be developed rigorously (see [62] and references therein).…”

Medium Access Control protocol for Collaborative Spectrum Learning in Wireless Networks

“…These variants aim to achieve orthogonal allocations in a fully-distributed manner without any information sharing between nodes. Some of these variants aim to achieve optimal OFDMA allocations where the sum of the Quality of Service (QoS) of all the users is maximized [18], [19], [20], [25], [26], [27] while others aim to achieve stable OFDMA allocations where no user can unilaterally improve its QoS [22], [23], [24], [29]. Recently, new approaches incorporating fairness have also been proposed [21], [30].…”

“…In [37] DRL is used to maximize the total throughput or fairness when neighboring networks are using unknown MAC protocols such Aloha or TDMA. In [29] the authors achieve a stable allocation for the Restless MUMAB problem with heterogeneous channels. In [38] DQL for Markovian channels are considered where ε-greedy policies are used to accumulate the training data.…”