Multi-UAV Collision Avoidance using Multi-Agent Reinforcement Learning with Counterfactual Credit Assignment

Abstract

Multi-UAV collision avoidance is a challenging task for UAV swarm applications due to the need of tight cooperation among swarm members for collision-free path planning. Centralized Training with Decentralized Execution (CTDE) in Multi-Agent Reinforcement Learning is a promising method for multi-UAV collision avoidance, in which the key challenge is to effectively learn decentralized policies that can maximize a global reward cooperatively. We propose a new multi-agent critic-actor learning scheme called MACA for UAV swarm collision avoidance. MACA uses a centralized critic to maximize the discounted global reward that considers both safety and energy efficiency, and an actor per UAV to find decentralized policies to avoid collisions. To solve the credit assignment problem in CTDE, we design a counterfactual baseline that marginalizes both an agent's state and action, enabling to evaluate the importance of an agent in the joint observation-action space. To train and evaluate MACA, we design our own simulation environment MACAEnv to closely mimic the realistic behaviors of a UAV swarm. Simulation results show that MACA achieves more than 16% higher average reward than two state-of-the-art MARL algorithms and reduces failure rate by 90% and response time by over 99% compared to a conventional UAV swarm collision avoidance algorithm in all test scenarios.

A Comprehensive Survey on Multi-Agent Reinforcement Learning for Connected and Automated Vehicles

“…In the lazy agent problem, the agents in a team are not performing equally well, but they are still receiving the same collective reward. To address this issue, researchers have suggested various learning and non-learning methods that assign credit to each agent based on their individual contributions [ 30 , 31 , 32 , 33 , 34 , 35 ]. Interestingly, the centralized training and decentralized execution indigenously have no issues or occurrences of lazy agents.…”

A Concept of Autonomous Multi-Agent Navigation System for Unmanned Surface Vessels

“…The last column contains additional remarks, concerning advantages, limitations or other comments with regard to the relation between the mentioned methods and the approach described in this paper. Some of the proposed multi-agent systems are intended for solving a different task than collision avoidance, considered in this paper, such as formation control [22,23], training a team of USVs [24], searching water region by a team of UUV [25] and/or consider a different type of vehicle, such as UAV [26], UUV or a car [27].…”

“…The concept of the maneuver auto-negotiation system in relation to autonomous cars was proposed in [27], to UAV in [26] and to ships in [28][29][30]. In [28], collision avoidance and maneuver auto-negotiation is based on geometrical relationships.…”