A Distributed Multi-Agent RL-Based Autonomous Spectrum Allocation Scheme in D2D Enabled Multi-Tier HetNets

Abstract

Multi-tier heterogeneous networks (HetNets) and device-to-device (D2D) communication are vastly considered in 5G networks. The interference mitigation and resource allocation in the D2D enabled multi-tier HetNets is a cumbersome and challenging task that cannot be solved by the conventional centralized resource allocation techniques proposed in the literature. In this paper, we propose a distributed multi-agent learning-based spectrum allocation scheme in which D2D users learn the wireless environment and select spectrum resources autonomously to maximize their throughput and spectral efficiency (SE) while causing minimum interference to the cellular users. We have employed the distributed learning in a stochastic geometry-based realistic multi-tier heterogeneous network to validate the performance of our scheme. The proposed scheme enables the D2D users to achieve higher throughput and SE, higher signal-to-interferenceplus-noise ratio and low outage ratio for cellular users, and better computational time efficiency and performs well in the dense multi-tier HetNets without affecting network coverage compared with the distance based resource criterion and joint-resource allocation and link adaptation schemes.INDEX TERMS D2D communication, multi-agent reinforcement learning, autonomous spectrum allocation, distributed reinforcement learning, heterogeneous networks, interference mitigation in D2D enabled HetNets.

Integration of D2D, Network Slicing, and MEC in 5G Cellular Networks: Survey and Challenges

“…A lot of work has been done in the recent past to solve the resource management problems in D2D using intelligent decision making and RL methods [105]- [108]. The work in [106] used the Q learning to maximize the system throughput, [107] used distributed method for spectrum allocation considering interference to maximize the network throughput, [108] proposed the intelligent resource allocation in D2D based vehicular networks using RL technique called as transfer actor-critic (AC). Currently, the concept of deep learning has been investigated for resource allocation scenarios [109]- [112].…”

D2D Mobile Relaying Meets NOMA—Part II: A Reinforcement Learning Perspective

“…The authors of [ 18 ] came up with a distributed RL algorithm that dynamically changes the transmitted data and power control at each sensor node according to observed state information such that the data of all sensor nodes are received while minimizing the delay. The authors in [ 21 ] came up with a distributed RL-based resource allocation scheme to mitigate interference between D2D users and cellular users. A distributed spectrum allocation framework based on multi-agent deep reinforcement learning was also proposed in [ 22 ].…”

D2D communication for spectral efficiency improvement and interference reduction: A survey

“…The main disadvantage of this algorithm is that the network model considers only one macro cell and only one user is connected to a single femtocell, which is impractical. In [47], an autonomous resource allocation scheme using an autonomous Q-learning algorithm was develop to improve the spectrum efficiency and control the interference in D2D-enabled multi-tier HetNets. This algorithm aims to autonomously reduce the load on the BS and the spectrum using the devices and to increase the throughput of D2D users while maintaining the QoS requirements and outage ratio of CUs.…”

“…The scheme proposed by [46] is impractical because it only uses one macrocell and only one user is connected to a single femtocell. The remaining systems proposed by [11,[47][48][49] either cause synchronization loss, unreliability, require network assistance, or increase complexity.…”