Energy Efficient 3-D UAV Control for Persistent Communication Service and Fairness: A Deep Reinforcement Learning Approach

Qi, Hu; Hu, Zhiqun; Huang, Hao; Wen, Xiangming; Lu, Zhaoming

doi:10.1109/access.2020.2981403

Cited by 65 publications

(63 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this scheme, the UAV-BS can intelligently allocate uplink channels and the transmit power of IoT nodes for maximizing the energy performance of all IoT nodes. Another UAV control policy based on DRL called the deep deterministic policy gradient (UC-DDPG) was proposed in [35]. UC-DDPG addressed the combined problem of 3D mobility of multiple UAVs and energy recharging arrangements to ensure efficient energy and fair broad region coverage of each user with keeping on the service.…”

Section: Literature Reviewmentioning

confidence: 99%

Gateway Selection in Millimeter Wave UAV Wireless Networks Using Multi-Player Multi-Armed Bandit

Mohamed

Hashima

Aldosary

et al. 2020

Sensors

View full text Add to dashboard Cite

Recently, unmanned aerial vehicle (UAV)-based communications gained a lot of attention due to their numerous applications, especially in rescue services in post-disaster areas where the terrestrial network is wholly malfunctioned. Multiple access/gateway UAVs are distributed to fully cover the post-disaster area as flying base stations to provide communication coverage, collect valuable information, disseminate essential instructions, etc. The access UAVs after gathering/broadcasting the necessary information should select and fly towards one of the surrounding gateways for relaying their information. In this paper, the gateway UAV selection problem is addressed. The main aim is to maximize the long-term average data rates of the UAVs relays while minimizing the flights’ battery cost, where millimeter wave links, i.e., using 30~300 GHz band, employing antenna beamforming, are used for backhauling. A tool of machine learning (ML) is exploited to address the problem as a budget-constrained multi-player multi-armed bandit (MAB) problem. In this setup, access UAVs act as the players, and the arms are the gateway UAVs, while the rewards are the average data rates of the constructed relays constrained by the battery cost of the access UAV flights. In this decentralized setting, where information is neither prior available nor exchanged among UAVs, a selfish and concurrent multi-player MAB strategy is suggested. Towards this end, three battery-aware MAB (BA-MAB) algorithms, namely upper confidence bound (UCB), Thompson sampling (TS), and the exponential weight algorithm for exploration and exploitation (EXP3), are proposed to realize gateways selection efficiently. The proposed BA-MAB-based gateway UAV selection algorithms show superior performance over approaches based on near and random selections in terms of total system rate and energy efficiency.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Gateway Selection in Millimeter Wave UAV Wireless Networks Using Multi-Player Multi-Armed Bandit

Mohamed

Hashima

Aldosary

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Responding to the needs, there have been various studies and algorithms developed for autonomous flight systems. Especially, many ML-based (Machine-learning based) methods have been proposed for autonomous path finding [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ]. However, they are limited when applied to a large target area.…”

Section: Introductionmentioning

confidence: 99%

“…For realistic driving scenarios, DDPG (Deep Deterministic Policy Gradient) [ 13 ] adapts the ideas of DQN to the continuous action domain. Many extended studies of DDPG have been developed for various applications [ 14 , 15 , 16 , 17 ]. Kong et al [ 14 ] used state-adversarial deep deterministic policy gradient algorithm (SA-DDPG) for combat maneuver decisions of an opponent aircraft being considered assuming gun-based aerial combat WVR.…”

Section: Introductionmentioning

confidence: 99%

“…Unlike existing Q-learning based methods, Gupta et al [ 15 ] took the learning loss into account under deterministic as well as stochastic policy gradient by combining Variation Autoencoder (VAE), Deep Deterministic Policy Gradient (DDPG), and Soft Actor–Critic (SAC). Qi et al [ 16 ] presented a UAV control policy based on DDPG to address the combination problem of 3D mobility of multiple UAVs and energy replenishment scheduling. In addition, a MEP–DDPG algorithm was designed using model predictive control and simulated annealing to generate expert experiences [ 17 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Fisheye-Based Smart Control System for Autonomous UAV Operation

Han¹

2020

Sensors

View full text Add to dashboard Cite

Recently, as UAVs (unmanned aerial vehicles) have become smaller and higher-performance, they play a very important role in the Internet of Things (IoT). Especially, UAVs are currently used not only in military fields but also in various private sectors such as IT, agriculture, logistics, construction, etc. The range is further expected to increase. Drone-related techniques need to evolve along with this change. In particular, there is a need for the development of an autonomous system in which a drone can determine and accomplish its mission even in the absence of remote control from a GCS (Ground Control Station). Responding to such requirements, there have been various studies and algorithms developed for autonomous flight systems. Especially, many ML-based (Machine-Learning-based) methods have been proposed for autonomous path finding. Unlike other studies, the proposed mechanism could enable autonomous drone path finding over a large target area without size limitations, one of the challenges of ML-based autonomous flight or driving in the real world. Specifically, we devised Multi-Layer HVIN (Hierarchical VIN) methods that increase the area applicable to autonomous flight by overlaying multiple layers. To further improve this, we developed Fisheye HVIN, which applied an adaptive map compression ratio according to the drone’s location. We also built an autonomous flight training and verification platform. Through the proposed simulation platform, it is possible to train ML-based path planning algorithms in a realistic environment that takes into account the physical characteristics of UAV movements.

show abstract

“…In [17,18], multiple decision variables in the problem modelings may lead to huge action space and slow convergence (more than 1000 learning episodes). It is noted that the solution proposed in [17,18] can be applied to only unconstrained problems. However, for general UAV-assisted networks, the optimization problems have constraints [4][5][6][7][8][9][11][12][13].…”

Section: Introductionmentioning

confidence: 99%

Actor-Critic Learning Based Energy Optimization for UAV Access-and-Backhaul Networks

Yuan

Lei

et al. 2020

Preprint

View full text Add to dashboard Cite

In unmanned aerial vehicle ( UAV )-assisted networks, UAV acts as an aerial base station to serve ground users ( GUs ) through an access network, and meanwhile, the requested data traffic is acquired through the backhaul link. In this paper, we investigate an energy minimization problem with limited power supply for both backhaul and access links. The difficulties for solving such a non-convex and combinatorial problem lie at the high computational complexity/time. In solution development, we consider the approaches from both actor-critic deep reinforcement learning (AC- DRL ) and optimization perspectives. Firstly, two offline non-learning algorithms, i.e., an optimal and a heuristic algorithms, based on piecewise linear approximation and relaxation are developed as benchmarks. Secondly, towards real-time decision making, we improve the conventional AC- DRL and propose two learning schemes: AC-based user group scheduling and backhaul power allocation ( ACGP ), and joint AC-based user group scheduling and optimization-based backhaul power allocation ( ACGOP ). Numerical results show that the computation time of both ACGP and ACGOP are reduced tenfold to hundredfold compared to the offline approaches, where ACGOP is better than ACGP in energy savings. The results also verify the superiority of proposed learning solutions in terms of guaranteeing the feasibility and minimizing the system energy compared to the conventional AC- DRL .

show abstract

Energy Efficient 3-D UAV Control for Persistent Communication Service and Fairness: A Deep Reinforcement Learning Approach

Cited by 65 publications

References 58 publications

Gateway Selection in Millimeter Wave UAV Wireless Networks Using Multi-Player Multi-Armed Bandit

Gateway Selection in Millimeter Wave UAV Wireless Networks Using Multi-Player Multi-Armed Bandit

Fisheye-Based Smart Control System for Autonomous UAV Operation

Actor-Critic Learning Based Energy Optimization for UAV Access-and-Backhaul Networks

Contact Info

Product

Resources

About