A 3D Game Theoretical Framework for the Evaluation of Unmanned Aircraft Systems Airspace Integration Concepts

Abstract

Predicting the outcomes of integrating Unmanned Aerial Systems (UAS) into the National Airspace System (NAS) is a complex problem which is required to be addressed by simulation studies before allowing the routine access of UAS into the NAS. This paper focuses on providing a 3-dimensional (3D) simulation framework using a game theoretical methodology to evaluate integration concepts using scenarios where manned and unmanned air vehicles co-exist. In the proposed method, human pilot interactive decision making process is incorporated into airspace models which can fill the gap in the literature where the pilot behavior is generally assumed to be known a priori. The proposed human pilot behavior is modeled using dynamic levelk reasoning concept and approximate reinforcement learning. The level-k reasoning concept is a notion in game theory and is based on the assumption that humans have various levels of decision making. In the conventional "static" approach, each agent makes assumptions about his or her opponents and chooses his or her actions accordingly. On the other hand, in the dynamic level-k reasoning, agents can update their beliefs about their opponents and revise their level-k rule. In this study, Neural Fitted Q Iteration, which is an approximate reinforcement learning method, is used to model time-extended decisions of pilots with 3D maneuvers. An analysis of UAS integration is conducted using an example 3D scenario in the presence of manned aircraft and fully autonomous UAS equipped with sense and avoid algorithms.

Modeling cyber-physical human systems via an interplay between reinforcement learning and game theory

“…This result was a continuation of leading studies conducted by Oyler et al (2016) and Li et al (2016). Similar to Musavi et al (2016) and Musavi et al (2018), this contribution is also the first in the automotive literature where a large number of decision making drivers are simultaneously modeled using a game theoretical modeling approach. Recently, an extended version of this work, which covers a larger class of interaction scenarios, with the help of a road traffic simulation on a 5-lane highway, is presented in Albaba et al (2019), where validation studies by processing real traffic data, which is provided in Colyar and Halkias (2007), is conducted.…”

“…Although it is demonstrated that the proposed framework can be used to provide significant qualitative analysis power in UAS integration scenarios, the study is still limited and need to be extended to 3D airspace. Preliminary studies are conducted in this direction, details of which can be found in Musavi et al (2018). One important distinction in this study is the need for an approximate reinforcement learning algorithm to handle the dramatically increased state space due to the 3D geometry.…”

“…In this research, Bayesian Networks are not utilized and the probabilistic decision making is obtained through employing a stochastic reinforcement learning algorithm proposed in Jaakkola et al (1994). Recently, this work is extended for the cases where the aircraft can move both horizontally and vertically in a 3-dimensional airspace (Musavi et al, 2018). This extension required a dramatically larger observation space for the pilots which ruled out the possibility of using exact RL methods.…”

The Power of d Choices in Scheduling for Data Centers with Heterogeneous Servers

“…This method is out of the scope of this paper, but is a promising future work to be combined with our proposed algorithm. As a complementary method, reinforcement learning tools can be utilized to model the data center behavior to estimate the processing rates of task types on servers as used in scenarios where both manned and unmanned vehicles coexist [23], [24], [25], [26], [27], and [28].…”