Deep Reinforcement Learning for Dynamic Multichannel Access in Multi-Cognitive Radio Networks

Abstract

We propose a reinforcement learning framework based on deep recurrent learning to solve the dynamic spectrum access problem in the scenario where multiple cognitive networks coexist. In this scenario, the shared spectrum is divided into multiple channels, and the channel occupation by the primary user is modeled as a Markov model. The observation of the channel status by secondary users in this area obeys the partially observed Markov process, that is, each user can only observe the status of one channel in each time slot, and cannot obtain global information. To avoid collision in different cognitive networks, we adopt the distributed multi-agent deep reinforcement learning method. Without the interaction of different cognitive networks, based on the current partial observations, a suitable channel access policy can be obtained by a neural network. Comparing with Slotted-aloha and DQN-DSA Algorithm, the result indicates DRQN-DSA proposed in this paper performs better.

Channel Allocation to GAA Users Using Double Deep Recurrent Q-Learning Based on Double Auction Method

“…Authors in [34] proposed the concept of the Recurrent neural network (RNN) to solve the computational games for partial observations based on MDPs called (POMDP). The authors in [35]- [37] integrated the LSTM with RNN to solve allocation problems in the DSA to maintain the sequence information along with the internal states. The authors investigated the proposed DSA scenario in absence of the primary users and developed the DRQN for the opportunistic users to learn only good policies which is not a practical approach adopted by the DSA in 5G and beyond communications.…”

Double Deep Recurrent Reinforcement Learning for Centralized Dynamic Multichannel Access

“…Recurrent neural network (RNN) is proposed to solve such partial observation Markov decision problems (POMDPs) in computer games [13,14]. Based on this advantage of RNN, Long Short-Term Memory (LSTM) layer (a RNN layer) is added to the neural network for solving such DSA problems, which can maintain an internal state and integrate sequence information [15][16][17]. Naparstek and Cohen consider a distributed DSA environment where each SU develops Deep Recurrent Q-Network (DRQN) to learn good policies [15].…”

“…However, this work assumes that there is no PUs in the scenario, which only represent specific situation and is difficult to apply to current radio environment where most spectrum resources have already been allocated. Different from [15][16][17] consider a more complex scenario where multiple PUs and multiple SUs coexist and there is no information interaction between the SUs.…”