Model-aided Deep Reinforcement Learning for Sample-efficient UAV Trajectory Design in IoT Networks

Abstract

Deep Reinforcement Learning (DRL) is gaining attention as a potential approach to design trajectories for autonomous unmanned aerial vehicles (UAV) used as flying access points in the context of cellular or Internet of Things (IoT) connectivity. DRL solutions offer the advantage of on-the-go learning hence relying on very little prior contextual information. A corresponding drawback however lies in the need for many learning episodes which severely restricts the applicability of such approach in real-world time-and energy-constrained missions.Here, we propose a model-aided deep Q-learning approach that, in contrast to previous work, considerably reduces the need for extensive training data samples, while still achieving the overarching goal of DRL, i.e to guide a battery-limited UAV towards an efficient data harvesting trajectory, without prior knowledge of wireless channel characteristics and limited knowledge of wireless node locations. The key idea consists in using a small subset of nodes as anchors (i.e. with known location) and learning a model of the propagation environment while implicitly estimating the positions of regular nodes. Interaction with the model allows us to train a deep Q-network (DQN) to approximate the optimal UAV control policy. We show that in comparison with standard DRL approaches, the proposed model-aided approach requires at least one order of magnitude less training data samples to reach identical data collection performance, hence offering a first step towards making DRL a viable solution to the problem.

Reinforcement Learning for Joint Detection and Mapping Using Dynamic UAV Networks

“…‚ Model-aided Deep Reinforcement Learning (Q Learning) [31] ‚ MARL with Deep Q-Learning [32] approaches, RL and deep RL have been used for UAV policy navigation because of their ability to learn directly by interactions with the surrounding environments [24], [39]- [43]. When the environment has a grid-world representation (e.g., indoors), Q-learning represents a simple and optimal solution because state-action pairs can be represented by a tractable Q-table that is updated at each time instant according to the received rewards [4], [14], [44].…”

On collaborative multi-UAV trajectory planning for data collection

“…Until now, very few studies have been reported to deal exclusively with the UAV trajectory design problem by considering the partially observed networking environment and unexpected mobility/locations of the IoT devices. In [20], the authors introduced a model-based deep reinforcement learning (DRL) UAV path planning algorithm for data collection, where a device localization mechanism was used by dividing the ground nodes into either known or unknown locations. Nonetheless, they made the assumption that the UAVs are given predetermined targets and the IoT nodes are static with complete location information.…”