Reinforcement Learning in Multiple-UAV Networks: Deployment and Movement Design

Abstract

A novel framework is proposed for quality of experience (QoE)-driven deployment and dynamic movement of multiple unmanned aerial vehicles (UAVs). The problem of joint non-convex three-dimensional (3D) deployment and dynamic movement of the UAVs is formulated for maximizing the sum mean opinion score (MOS) of ground users, which is proved to be NP-hard. In the aim of solving this pertinent problem, a three-step approach is proposed for attaining 3D deployment and dynamic movement of multiple UAVs. Firstly, a genetic algorithm based K-means (GAK-means) algorithm is utilized for obtaining the cell partition of the users. Secondly, Q-learning based deployment algorithm is proposed, in which each UAV acts as an agent, making their own decision for attaining 3D position by learning from trial and mistake. In contrast to the conventional genetic algorithm based learning algorithms, the proposed algorithm is capable of training the direction selection strategy offline. Thirdly, Q-learning based movement algorithm is proposed in the scenario that the users are roaming. The proposed algorithm is capable of converging to an optimal state. Numerical results reveal that the proposed algorithms show a fast convergence rate after a small number of iterations. Additionally, the proposed Q-learning based deployment algorithm outperforms K-means algorithms and Iterative-GAKmean (IGK) algorithms with low complexity.Index Terms-Dynamic movement, Q-learning, quality of experience (QoE), three-dimensional deployment, unmanned aerial vehicle (UAV)

A Unified Spatial Framework for UAV-Aided MmWave Networks

“…The optimal strategies have been provided in various UAV applications, such as securing UAV communications [22], [23], multi-hop UAV relaying communications [24], and so forth. In addition, promising machine learning approaches have been adopted to design the movement and power control in multi-UAV assisted networks [25], [26].…”

A Reinforcement Learning Approach for Fair User Coverage Using UAV Mounted Base Stations Under Energy Constraints

“…The above proposed method was extended to a distributed DRL-based control solution in [27]. In [28], a threefold solution was proposed for using UAV-BSs to provide coverage to ground users. First, the authors proposed using genetic algorithm based K-means clustering (GAK-means) to partition the users into cells.…”

Non-Cooperative Aerial Base Station Placement via Stochastic Optimization

“…A key idea in the proposed framework is to sidestep these difficulties by capitalizing on stochastic optimization methods. These methods stem from the observation that ∇J(l) in (8) can be expressed as ∇J(l) = E{∇J m (l)} and replaced with an estimate, as done for J(l) in (6). The idea is to update l every time an MU (or a certain number of MUs) sends the relevant information through the control channel.…”

“…The final arrangement meets the target QoS at 198 out of the 202 MUs. As a benchmark, the histogram is compared with the one obtained if the AirBSs used K-means, which is the algorithm underlying the approaches in [6] and [17]. K-means performs poorly here because the two users off the area of interest shift the centroids.…”

“…in [2]- [5]. To accommodate multiple AirBSs, centralized schemes have been proposed in [6]- [9], but they require a central controller that receives all system information in real time and instructs the AirBSs on how to navigate, which may be problematic in practice due to the unreliable nature of wireless communications. To alleviate this limitation, decentralized schemes that require communication and coordination only among neighboring AirBSs as well as between AirBSs and MUs have been developed in [10], Second, the designed utility is maximized invoking tools from stochastic optimization, which constitutes the workhorse of deep neural network training given their simplicity and low computational requirements.…”