Multi-agent hierarchical policy gradient for Air Combat Tactics emergence via self-play

Sun, Zhimei; Piao, Haiyin; Yang, Zhen; Zhao, Yiyang; Zhan, Guang; Zhou, Deyun; Meng, Guang; Chen, Hechang; Chen, Xing; Qu, Bohao; Lu, Yuanjie

doi:10.1016/j.engappai.2020.104112

Cited by 68 publications

(27 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [12], a voltage control for a power grid [12] is designed, and MARL is applied with success for route planning in road network environments in [13]. MARL is also used for training unmanned fighter aircrafts in air-to-air combat in [14]. RL for HVAC systems has previously been studied.…”

Section: Action-state Space Sizementioning

confidence: 99%

A Multi-Agent Reinforcement Learning Approach to Price and Comfort Optimization in HVAC-Systems

2021

View full text Add to dashboard Cite

This paper addresses the challenge of minimizing training time for the control of Heating, Ventilation, and Air-conditioning (HVAC) systems with online Reinforcement Learning (RL). This is done by developing a novel approach to Multi-Agent Reinforcement Learning (MARL) to HVAC systems. In this paper, the environment formed by the HVAC system is formulated as a Markov Game (MG) in a general sum setting. The MARL algorithm is designed in a decentralized structure, where only relevant states are shared between agents, and actions are shared in a sequence, which are sensible from a system’s point of view. The simulation environment is a domestic house located in Denmark and designed to resemble an average house. The heat source in the house is an air-to-water heat pump, and the HVAC system is an Underfloor Heating system (UFH). The house is subjected to weather changes from a data set collected in Copenhagen in 2006, spanning the entire year except for June, July, and August, where heat is not required. It is shown that: (1) When comparing Single Agent Reinforcement Learning (SARL) and MARL, training time can be reduced by 70% for a four temperature-zone UFH system, (2) the agent can learn and generalize over seasons, (3) the cost of heating can be reduced by 19% or the equivalent to 750 kWh of electric energy per year for an average Danish domestic house compared to a traditional control method, and (4) oscillations in the room temperature can be reduced by 40% when comparing the RL control methods with a traditional control method.

show abstract

Section: Action-state Space Sizementioning

confidence: 99%

A Multi-Agent Reinforcement Learning Approach to Price and Comfort Optimization in HVAC-Systems

2021

View full text Add to dashboard Cite

show abstract

“…In modern air warfare, fighters usually perform combat missions information, with corresponding tactics laid out through the command system. These tactics are timevarying and have different characteristics, which brings a lot of uncertainty to target tactical intention recognition [15]. To improve accuracy, more parameters are integrated to build the recognition model.…”

Section: General Technology Schemementioning

confidence: 99%

Target Tactical Intention Recognition in Multiaircraft Cooperative Air Combat

Meng

Zhao

Wang

et al. 2021

International Journal of Aerospace Engineering

View full text Add to dashboard Cite

Accurately identifying the tactical intention of the target can facilitate the prediction of the opponent’s behavior and improve the efficiency of collaborative decision. We have observed that traditional methods could achieve high recognition rate on conventional tactical intent. Nevertheless, their performance would deteriorate seriously when recognizing cooperative tactical intention in multiaircraft air combat environment. The main reason resides on key features that are difficult to extract for traditional methods. To this end, this paper proposes a novel approach to recognizing tactical intention of multiaircraft cooperative air combat. Specifically, we employ support vector machine (SVM) to forecast the attack intention based on 19 low correlation features. The purpose of the employment of SVM is to avoid local optimization and reduce data dimension. Moreover, we use three models, i.e., dynamic Bayesian network (DBN), radar model, and threat assessment model to extract crucial information regarding maneuver occupancy, silent penetration, and attack tendency. The extracted information would make great contribution to the recognition accuracy of six types of cooperative tactics. Finally, we learn a decision tree model on train samples processed by above two phases to classify different tactical intention. In order to verify the effectiveness of the proposed method, we use data sets from a loop simulation platform. The experimental results have approved the superiority of our method via the comparison to several baseline methods with respect to recognition rate and efficiency. In addition, we underline that our method also performs well on incomplete and uncertain information.

show abstract

“…Multi-robotic cooperation has received extensive research attention in diverse domains due to its efficiency and capability in conducting complex missions [1], e.g., cooperative surveillance [2], search and rescue [3], and air combat [4], etc. As a typical application scenario, the cooperative pursuit aims to coordinate multiple robots for capturing one or multiple evaders [5] in a confined environment with obstacles.…”

Section: Introductionmentioning

confidence: 99%

Multi-robot Cooperative Pursuit via Potential Field-Enhanced Reinforcement Learning

Zhang¹,

Wang²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

It is of great challenge, though promising, to coordinate collective robots for hunting an evader in a decentralized manner purely in light of local observations. In this paper, this challenge is addressed by a novel hybrid cooperative pursuit algorithm that combines reinforcement learning with the artificial potential field method. In the proposed algorithm, decentralized deep reinforcement learning is employed to learn cooperative pursuit policies that are adaptive to dynamic environments. The artificial potential field method is integrated into the learning process as predefined rules to improve the data efficiency and generalization ability. It is shown by numerical simulations that the proposed hybrid design outperforms the pursuit policies either learned from vanilla reinforcement learning or designed by the potential field method. Furthermore, experiments are conducted by transferring the learned pursuit policies into real-world mobile robots. Experimental results demonstrate the feasibility and potential of the proposed algorithm in learning multiple cooperative pursuit strategies.

show abstract

Multi-agent hierarchical policy gradient for Air Combat Tactics emergence via self-play

Cited by 68 publications

References 14 publications

A Multi-Agent Reinforcement Learning Approach to Price and Comfort Optimization in HVAC-Systems

A Multi-Agent Reinforcement Learning Approach to Price and Comfort Optimization in HVAC-Systems

Target Tactical Intention Recognition in Multiaircraft Cooperative Air Combat

Multi-robot Cooperative Pursuit via Potential Field-Enhanced Reinforcement Learning

Contact Info

Product

Resources

About