Cooperative Multiagent Deep Deterministic Policy Gradient (CoMADDPG) for Intelligent Connected Transportation with Unsignalized Intersection

Abstract

Unsignalized intersection control is one of the most critical issues in intelligent transportation systems, which requires connected and automated vehicles to support more frequent information interaction and on-board computing. It is very promising to introduce reinforcement learning in the unsignalized intersection control. However, the existing multiagent reinforcement learning algorithms, such as multiagent deep deterministic policy gradient (MADDPG), hardly handle a dynamic number of vehicles, which cannot meet the need of the real road condition. Thus, this paper proposes a Cooperative MADDPG (CoMADDPG) for connected vehicles at unsignalized intersection to solve this problem. Firstly, the scenario of multiple vehicles passing through an unsignalized intersection is formulated as a multiagent reinforcement learning (RL) problem. Secondly, MADDPG is redefined to adapt to the dynamic quantity agents, where each vehicle selects reference vehicles to construct a partial stationary environment, which is necessary for RL. Thirdly, this paper incorporates a novel vehicle selection method, which projects the reference vehicles on a virtual lane and selects the largest impact vehicles to construct the environment. At last, an intersection simulation platform is developed to evaluate the proposed method. According to the simulation result, CoMADDPG can reduce average travel time by 39.28% compared with the other optimization-based methods, which indicates that CoMADDPG has an excellent prospect in dealing with the scenario of unsignalized intersection control.

A Traffic-Aware Federated Imitation Learning Framework for Motion Control at Unsignalized Intersections with Internet of Vehicles

“…Guan et al [6] proposed an RL-based method for guiding a fixed number of vehicles through an intersection. Wu et al [7] proposed a cooperative RL method to improve traffic efficiency while ensuring safety. This method decoupled the relationship between identity and driving information on vehicles.…”

A Comprehensive Survey on Cooperative Intersection Management for Heterogeneous Connected Vehicles

“…Qian et al [254] developed an IM technique based on two mixed-integer nonlinear programs where vehicle trajectories, departure and signal times were centrally optimized so that safety was ensured and travel times were shortened for CAVs. In addition, Wu et al [273] introduced a cooperative distributed collision-free framework for CAVs by formulating the scenario as a multi-agent RL problem. The proposed method could manage dynamic quantity of vehicles to pass through the intersection with efficient trip time.…”

Collaborative Autonomous Driving—A Survey of Solution Approaches and Future Challenges

“…Despite the rapid and progressive advancements in intersection management systems, we highlight some of the open issues and challenges of CIMs that need more research and advanced solution approaches to overcome these issues. Some of the common challenges are optimized resource utilization, several uncertainties from the control, communication, mechanical perspectives, vulnerable road user detection, sensing and identification of pedestrian and cyclists at non-signalized intersections, communication issues, such as delays packet loss, security, bandwidth limitations, low-quality data, distributed processing of data either at intersection controller or by CAV, and collision avoidance [ 46 , 50 ].…”