A Double Q-Learning Approach for Navigation of Aerial Vehicles with Connectivity Constraint

Abstract

This paper studies the trajectory optimization problem for an aerial vehicle with the mission of flying between a pair of given initial and final locations. The objective is to minimize the travel time of the aerial vehicle ensuring that the communication connectivity constraint required for the safe operation of the aerial vehicle is satisfied. We consider two different criteria for the connectivity constraint of the aerial vehicle which leads to two different scenarios. In the first scenario, we assume that the maximum continuous time duration that the aerial vehicle is out of the coverage of the ground base stations (GBSs) is limited to a given threshold. In the second scenario, however, we assume that the total time periods that the aerial vehicle is not covered by the GBSs is restricted. Based on these two constraints, we formulate two trajectory optimization problems. To solve these non-convex problems, we use an approach based on the double Q-learning method which is a model-free reinforcement learning technique and unlike the existing algorithms does not need perfect knowledge of the environment. Moreover, in contrast to the well-known Q-learning technique, our double Q-learning algorithm does not suffer from the over-estimation issue. Simulation results show that although our algorithm does not require prior information of the environment, it works well and shows near optimal performance.

Learning-Based UAV Trajectory Optimization with Collision Avoidance and Connectivity Constraints

“…Each UAV's state is composed of an observable information vector and an unobservable (hidden) information vector, s " rs o , s h s, where the observable state can be observed by other UAVs, while the unobservable state can not. In the global frame, observable state includes the UAV's position, velocity v " rv x , v y s, and radius r, i.e., s o " rp, v, rs P R 6 . The unobservable state consists of the destination p D , maximum speed v max , and orientation φ, i.e., s h " rp D , v max , φs P R 5 .…”

“…Trajectory optimization for cellular-connected UAVs has been investigated in the literature, in which a UAV has a mission of flying between a pair of given initial and final locations. The authors in [4]- [6] addressed the trajectory optimization problem with the goal to minimize the UAV's mission completion time. Particularly, the authors in [4] considered how to determine the optimal path for the UAV, subject to a quality of connectivity constraint in the GBS-to-UAV link specified by a minimum receive signal-to-noise ratio target.…”

“…Dynamic programming based approximate solution was proposed in this paper. The authors in [6] studied the trajectory optimization problem for a UAV with two different criteria for the connectivity constraint: 1) the maximum continuous time duration that the UAV is out of the coverage of the GBSs is limited to a given threshold; 2) the total time periods that the UAV is not covered by the GBSs is restricted. A double Q-learning method is proposed to solve the problem.…”