A Novel Framework for Dynamic Spectrum Management in MultiCell OFDMA Networks Based on Reinforcement Learning

Abstract

Abstract-In this work the feasibility of Reinforcement Learning (RL) for Dynamic Spectrum Management (DSM) in the context of next generation multicell Orthogonal Frequency Division Multiple Access (OFDMA) networks is studied. An RL-based algorithm is proposed and it is shown that the proposed scheme is able to dynamically find spectrum assignments per cell depending on the spatial distribution of the users over the scenario. In addition the proposed scheme is compared with other fixed and dynamic spectrum strategies showing the best tradeoff between spectral efficiency and Quality-of-Service (QoS).

An Application of Reinforcement Learning for Efficient Spectrum Usage in Next-Generation Mobile Cellular Networks

“…The first term in (3) performs the gradient ascent of the reward signal as in the learning algorithm considered in [17]- [19]. Parameter α(t) > 0 is called the learning rate (for details regarding its update see [17]). r i (t) is the average reward obtained asr i (t) = βr i (t) + (1 − β)r i (t − 1), with 0 < β 1.…”

“…Second, a novel model for practical implementation of the reward signal is given, extending and exploiting what was briefly described in [17]- [19]. Third, an exhaustive performance comparison with, fixed, hybrid, and other DSA strategies within the private commons scenario is given, showing remarkable improvements over the rest of strategies.…”

“…More recently, RL has been applied to spectrum sensing [15] or spectrum sharing [16] procedures in OFDMA-based networks from a secondary spectrum market but not from a primary operator perspective. On the other hand, in [17]- [19], we introduced a DSA framework with RL capabilities (in the following RL-DSA) for a multicell OFDMA packet RAN. The algorithm learns the most suitable spectrum assignment for a given set of cells in order to maximize a given reward, in accordance with certain cellular system's performance objective.…”

Distributed spectrum management based on reinforcement learning

“…Also, a centralized version of the RL-DSA algorithm is simulated [8]. It is expected that this centralized strategy outperforms distributed RL-DSA thanks to its global vision of the spectrum assignment, but self-organized systems, and in particular the distributed RL-DSA presented here, aim at approximate centralized performance while scalability and autonomous capabilities are given to the system.…”

“…Hence, RL has been successfully applied to spectrum sensing [6] or spectrum sharing [7] procedures in Cognitive Radio (CR). Also, in our previous work, we showed that RL can be used to implement centralized dynamic spectrum assignment (DSA) strategies for primary cellular networks [8].…”