Multi-Agent Reinforcement Learning in NOMA-aided UAV Networks for Cellular Offloading

Abstract

A novel framework is proposed for cellular offloading with the aid of multiple unmanned aerial vehicles (UAVs), while non-orthogonal multiple access (NOMA) technique is employed at each UAV to further improve the spectrum efficiency of the wireless network. The optimization problem of joint three-dimensional (3D) trajectory design and power allocation is formulated for maximizing the throughput. Since ground mobile users are considered as roaming continuously, the UAVs need to be re-deployed timely based on the movement of users. In an effort to solve this pertinent dynamic problem, a K-means based clustering algorithm is first adopted for periodically partitioning users. Afterward, a mutual deep Q-network (MDQN) algorithm is proposed to jointly determine the optimal 3D trajectory and power allocation of UAVs. In contrast to the conventional DQN algorithm, the MDQN algorithm enables the experience of multiagent to be input into a shared neural network to shorten the training time with the assistance of state abstraction. Numerical results demonstrate that: 1) the proposed MDQN algorithm is capable of converging under minor constraints and has a faster convergence rate than the conventional DQN algorithm in the multi-agent case; 2) The achievable sum rate of the NOMA enhanced UAV network is 23% superior to the case of orthogonal multiple access (OMA); 3) By designing the optimal 3D trajectory of UAVs with the aid of the MDON algorithm, the sum rate of the network enjoys 142% and 56% gains than that of invoking the circular trajectory and the 2D trajectory, respectively.

Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey

“…The Fig. 9: An UAV-aided communication network, where the satellite can provide location information for UAVs, and each UAV provides communication service to ground users and aims to maximize its own throughput [161], or overall system throughput [162].…”

“…In a user-intensive network with an overloaded BS, the UAV-aided cellular offloading technique can leverage aerial base stations (ABSs) carried UAVs to improve the network connectivity. The authors in [162] investigate the joint optimization of UAV trajectory design and power allocation in such a UAV-aided cellular offloading scenario. Each UAV has a single antenna and shares the same frequency band.…”

“…Each UAV has a single antenna and shares the same frequency band. Different from [161], in [162], the roaming users are grouped into different clusters, and each UAV is associated with one cluster.…”