A Generic Spatiotemporal Scheduling for Autonomous UAVs: A Reinforcement Learning-Based Approach

Abstract

Considerable attention has been given to leverage a variety of smart city applications using unmanned aerial vehicles (UAVs). The rapid advances in artificial intelligence can empower UAVs with autonomous capabilities allowing them to learn from their surrounding environment and act accordingly without human intervention. In this paper, we propose a spatiotemporal scheduling framework for autonomous UAVs using reinforcement learning. The framework enables UAVs to autonomously determine their schedules to cover the maximum of pre-scheduled events spatially and temporally distributed in a given geographical area and over a predetermined time horizon. The designed framework has the ability to update the planned schedules in case of unexpected emergency events. The UAVs are trained using the Q-learning (QL) algorithm to find effective scheduling plan. A customized reward function is developed to consider several constraints especially the limited battery capacity of the flying units, the time windows of events, and the delays caused by the UAV navigation between events. Numerical simulations show the behavior of the autonomous UAVs for various scenarios and corroborate the ability of QL to handle complex vehicle routing problems with several constraints. A comparison with an optimal deterministic solution is also provided to validate the performance of the learning-based solution.

Machine Learning Methods for UAV Flocks Management-A Survey

“…In [17], a spatiotemporal scheduling framework for autonomous UAVs using RL was presented. The framework enabled UAVs to determine their schedules autonomously to cover the maximum pre-scheduled events that were spatially and temporally distributed in a given geographical area and over a pre-determined time horizon.…”

“…Given the information about most successful approaches, we provide, in the following section, some guidelines to researchers attempting to select an appropriate ML method to solve a particular problem at hand. First of all we define some criteria to classify a given UAV partition communication Distributed [102] and task allocation rate Q-learning Kim and Morrison [55] Waypoint selection reward DRL Zhao et al [137] Task allocation for heterogeneous Reward and DRL UAVs in uncertain environment coverage Yang et al [126] Minimizing sum power for Transmission DRL UAV-enabled MEC network delay Bouhamed et al [17] Spatiotemporal scheduling coverage Q-learning Kurdi et al [61] UAV allocation to survivors Throughput Locust inspired Wang et al [118] Task allocation for path length GA heterogeneous targets Hu et al [48] Task allocation Overall reward Clustering + for multiple teams Hungarian alg. + ACO Kurdi et al [60] Coordinated task allocation Throughput Locust inspired Ye et al [128] Cooperative task assignment Mission execution GA against ground stationary targets time Luo et al [72] Pesticide spraying task allocation Total profit GA Kurdi et al [59] Task allocation Bacteria-inspired among deployed UAVs…”

Artificial Intelligence for UAV-Enabled Wireless Networks: A Survey

“…3) RL for scheduling and resource management Beyond path planning for smart UAVs, one can think about smart event scheduling for a drone network. In this context, the authors in [154] propose a spatio-temporal scheduling framework for autonomous UAVs. The proposed RL solution is model-free based and uses the popular Q-learning algorithm.…”

A UAV-Assisted Data Collection for Wireless Sensor Networks: Autonomous Navigation and Scheduling

“…In our context, RL can empower UAVs with sufficient intelligence to make local decisions and autonomously accomplish necessary tasks without requiring the support of a central unit or human involvement. In [35], we have designed a single-algorithm RL solution for routing autonomous agents however, without considering the data collection challenges, and the environment hurdles such as obstacles. In this study, we develop a generic autonomous navigation and scheduling approach using a combination of two RL-based frameworks for navigating and scheduling a UAV collecting data from multiple ground nodes with the objective of minimizing the data collection time.…”