Hierarchical Deep Reinforcement Learning for Backscattering Data Collection With Multiple UAVs

Zhang, Yu; Mou, Zhiyu; Gao, Feifei; Xing, Ling; Jiang, Jing; Han, Zhu

doi:10.1109/jiot.2020.3024666

Cited by 86 publications

(47 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The focus here is on maximizing minimum throughput in a wirelessly powered network without a complex environment and navigation constraints, only for a single scenario at a time. Similarly, in [8] there is also a strong focus on the energy supply of IoT devices through backscatter communications when a team of UAVs collects their data. The authors propose a multi-agent approach that relies on the definition of ambiguous boundaries between clusters of sensors.…”

Section: A Related Workmentioning

confidence: 99%

Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning

Bayerlein

Theile

Caccamo

et al. 2021

IEEE Open J. Commun. Soc.

View full text Add to dashboard Cite

Harvesting data from distributed Internet of Things (IoT) devices with multiple autonomous unmanned aerial vehicles (UAVs) is a challenging problem requiring flexible path planning methods. We propose a multi-agent reinforcement learning (MARL) approach that, in contrast to previous work, can adapt to profound changes in the scenario parameters defining the data harvesting mission, such as the number of deployed UAVs, number, position and data amount of IoT devices, or the maximum flying time, without the need to perform expensive recomputations or relearn control policies. We formulate the path planning problem for a cooperative, non-communicating, and homogeneous team of UAVs tasked with maximizing collected data from distributed IoT sensor nodes subject to flying time and collision avoidance constraints. The path planning problem is translated into a decentralized partially observable Markov decision process (Dec-POMDP), which we solve through a deep reinforcement learning (DRL) approach, approximating the optimal UAV control policy without prior knowledge of the challenging wireless channel characteristics in dense urban environments. By exploiting a combination of centered global and local map representations of the environment that are fed into convolutional layers of the agents, we show that our proposed network architecture enables the agents to cooperate effectively by carefully dividing the data collection task among themselves, adapt to large complex environments and state spaces, and make movement decisions that balance data collection goals, flight-time efficiency, and navigation constraints. Finally, learning a control policy that generalizes over the scenario parameter space enables us to analyze the influence of individual parameters on collection performance and provide some intuition about system-level benefits.Index Terms-Internet of Things (IoT), map-based planning, multi-agent reinforcement learning (MARL), trajectory planning, unmanned aerial vehicle (UAV).

show abstract

Section: A Related Workmentioning

confidence: 99%

Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning

Bayerlein

Theile

Caccamo

et al. 2021

IEEE Open J. Commun. Soc.

View full text Add to dashboard Cite

show abstract

“…In [24], the authors propose a multi-agent DQL algorithm to maximize the minimum throughput by the joint optimization of path design and channel resource assignment in a uav-enabled wireless powered communication network, where the UAVs can fly according to a discrete set of actions. References [25] and [26] aim to minimize the overall flight time of the UAVs under their individual energy constraints. The authors in [25] develop an option-based hybrid DRL method that allows the UAV to choose between two algorithms to handle deterministic and ambiguous boundary scenarios.…”

Section: Related Workmentioning

confidence: 99%

“…References [25] and [26] aim to minimize the overall flight time of the UAVs under their individual energy constraints. The authors in [25] develop an option-based hybrid DRL method that allows the UAV to choose between two algorithms to handle deterministic and ambiguous boundary scenarios. However, the impact of obstacles on the navigation of the UAVs is ignored.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Deep Reinforcement Learning Based Energy Efficient Multi-UAV Data Collection for IoT Networks

Khodaparast

Wang

et al. 2021

IEEE Open J. Veh. Technol.

View full text Add to dashboard Cite

Unmanned aerial vehicles (UAVs) are regarded as an emerging technology, which can be effectively utilized to perform the data collection tasks in the Internet of Things (IoT) networks. However, both the UAVs and the sensors in these networks are energylimited devices, which necessitates an energy efficient data collection procedure to ensure the network lifetime. In this paper, we propose a multi-UAV-assisted network, where the UAVs fly to the ground sensors and control the sensor's transmit power during the data collection time. Our goal is to minimize the total energy consumption of the UAVs and the sensors, which is needed to accomplish the data collection mission. We formulate this problem into three sub-problems of single UAV navigation, sensor power control as well as multi-UAV scheduling and model each part as a finite-horizon Markov Decision Process (MDP). We deploy deep reinforcement learning (DRL)-based frameworks to solve each part. Specifically, we use deep deterministic policy gradient (DDPG) method to generate the best trajectory for the UAVs in an obstacle-constraint environment, given its starting position and the target sensor. We also deploy DDPG to control the sensor's transmit power during data collection. To schedule activity plans for each UAV to visit the sensors, we propose a multi-agent deep Q-learning (DQL) approach by taking the total energy consumption of the UAVs on each path into account. Our simulations show that the UAVs can find a safe and optimal path for each of their trips. Continuous power control of the sensors achieves better performance over the fixed power approaches in terms of the total energy consumption during data collection. In addition, compared to the two commonly used baselines, our scheduling framework achieves better and near-optimal results.

show abstract

“…Content may change prior to final publication. Citation information: DOI 10.1109/TGCN.2021.3095792, IEEE Transactions on Green Communications and Networking [122] Single-UAV assisted BackCom Design an optimal trajectory for UAV Achieved optimal trajectory and data collection and enable uplink multiple access high decoding of data at the UAV [123] Multiple-UAV assisted BackCom Design techniques for minimum Achieved shortest operation time when data collection operating time of UAV using DRL maximum (four) UAVs were deployed [124] UAV assisted throughput Design an operating protocol Achieved high throughput gain enhancement for UAV as a relay [125] D2D-relaying assisted throughput Design an optimal policy for setting Improved BackCom network improvement operating condition for relay nodes throughput [126] QoS guaranty in UAV-assisted Design techniques to ensure UAV relaying Achieved higher arrival rate from BackCom based IoT through BackCom and multiple access IoT nodes and higher number of to guaranty QoS connected IoT nodes [127] IRS assisted BackCom Jointly optimize phase shifts at Reduced transmitter power at the the IRS and transmitter beamforming source and enhanced the tag's range [128] Intelligent coherent signal Develop unsupervised learning algorithm Achieved coherent detection detection at receiver for joint signal estimation and decoding with a low-cost receiver [129] Intelligent signal detection Exploit the clustering phenomenon Achieved non-coherent signal at receiver…”

Section: F Technology-assisted and Intelligent Techniquesmentioning

confidence: 99%

Backscatter Wireless Communications and Sensing in Green Internet of Things

Toro¹,

Wu²,

Leung³

2021

Preprint

View full text Add to dashboard Cite

show abstract

Hierarchical Deep Reinforcement Learning for Backscattering Data Collection With Multiple UAVs

Cited by 86 publications

References 38 publications

Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning

Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning

Deep Reinforcement Learning Based Energy Efficient Multi-UAV Data Collection for IoT Networks

Backscatter Wireless Communications and Sensing in Green Internet of Things

Contact Info

Product

Resources

About