Reinforcement Learning-Based Flow Management Techniques for Urban Air Mobility and Dense Low-Altitude Air Traffic Operations

Panagiotakopoulos

2023

Drones

: The research community has paid great attention to the prediction of air traffic flows. Nonetheless, research examining the prediction of air traffic patterns for unmanned aircraft traffic management (UTM) is relatively sparse at present. Thus, this paper proposes a one-dimensional convolutional neural network and encoder-decoder LSTM framework to integrate air traffic flow prediction with the intrinsic complexity metric. This adapted complexity metric takes into account the important differences between ATM and UTM operations, such as dynamic flow structures and airspace density. Additionally, the proposed methodology has been evaluated and verified in a simulation scenario environment, in which a drone delivery system that is considered essential in the delivery of COVID-19 sample tests, package delivery services from multiple post offices, an inspection of the railway infrastructure and fire-surveillance tasks. Moreover, the prediction model also considers the impacts of other significant factors, including emergency UTM operations, static no-fly zones (NFZs), and variations in weather conditions. The results show that the proposed model achieves the smallest RMSE value in all scenarios compared to other approaches. Specifically, the prediction error of the proposed model is 8.34% lower than the shallow neural network (on average) and 19.87% lower than the regression model on average.

Section: Discussionmentioning

confidence: 99%

Deep Learning Architecture for UAV Traffic-Density Prediction

Alharbi

Panagiotakopoulos

2023

Drones

“…Other approaches such as hierarchical MARL also provide insight into DCB resolution for manned air traffic [15]. And for unmanned aircraft in UTM or UAM, the deep Q-Learning network (DQN) with genetic algorithm (GA) is a good attempt to resolve the congestion problem in urban lowaltitude airspace [16] consider weather changes. Because of the difference in airspace structure and operational notion, the hybrid operation of manned and unmanned flight, scheduled and on-demand flight, both pose a huge challenge for UAM.…”

Section: Introductionmentioning

confidence: 99%

Strategic Conflict Management using Recurrent Multi-agent Reinforcement Learning for Urban Air Mobility Operations Considering Uncertainties

Huang

Tsourdos

2023

J Intell Robot Syst

The rapidly evolving urban air mobility (UAM) develops the heavy demand for public air transport tasks and poses great challenges to safe and efficient operation in low-altitude urban airspace. In this paper, the operation conflict is managed in the strategic phase with multi-agent reinforcement learning (MARL) in dynamic environments. To enable efficient operation, the aircraft flight performance is integrated into the process of multi-resolution airspace design, trajectory generation, conflict management, and MARL learning. The demand and capacity balancing (DCB) issue, separation conflict, and block unavailability introduced by wind turbulence are resolved by the proposed the multi-agent asynchronous advantage actor-critic (MAA3C) framework, in which the recurrent actor-critic networks allow the automatic action selection between ground delay, speed adjustment, and flight cancellation. The learned parameters in MAA3C are replaced with random values to compare the performance of trained models. Simulated training and test experiments performed on a small urban prototype and various combined use cases suggest the superiority of the MAA3C solution in resolving conflicts with complicated wind fields. And the generalization, scalability, and stability of the model are also demonstrated while applying the model to complex environments.

“…The congestion problem caused by manned or unmanned aircraft in metropolitan regions, which is critical for air or ground operation efficiency and safety, is seldom investigated. There is only one effective attempt till now to apply reinforcement learning, specifically, a deep Q-Learning network (DQN) with a genetic algorithm (GA) to the UAM system and generate a feasible solution for the congestion problem [16].…”

Section: Introductionmentioning

confidence: 99%

Strategic Conflict Management for Performance-based Urban Air Mobility Operations with Multi-agent Reinforcement Learning

Huang

2022 International Conference on Unmanned Aircraft Systems (ICUAS)

Tsourdos

2022

With the urban air mobility (UAM) quickly evolving, the great demand for public airborne transit and deliveries, besides creating a big market, will result in a series of technical, operational, and safety problems. This paper addresses the strategic conflict issue in low-altitude UAM operations with multi-agent reinforcement learning (MARL). Considering the difference in flight characteristics, the aircraft performance is fully integrated into the design process of strategic deconfliction components. With this concept, the multi-resolution structure for the low-altitude airspace organization, Gaussian Mixture Model (GMM) for the speed profile generation, and dynamic separation minima enable efficient UAM operations. To resolve the demand and capacity balancing (DCB) issue and the separation conflict at the strategic stage, the multi-agent asynchronous advantage actor-critic (MAA3C) framework is built with mask recurrent neural networks (RNNs). Meanwhile, variable agent number, dynamic environments, heterogeneous aircraft performance, and action selection between speed adjustment and ground delay can be well handled. Experiments conducted on a developed prototype and various scenarios indicate the obvious advantages of the constructed MAA3C in minimizing the delay cost and refining speed profiles. And the effectiveness, scalability, and stabilization of the MARL solution are ultimately demonstrated.