CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenario

Zhang, Huichu; Feng, Siyuan; Liu, Chang; Ding, Yaoyao; Zhu, Yichen; Zhou, Zihan; Zhang, Weinan; Yu, Yong; Jin, Haiming; Li, Zhenhui

doi:10.1145/3308558.3314139

Cited by 167 publications

(73 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experiments are conducted on CityFlow 2 [32], an open-source traffic simulator that supports large-scale TSC. After providing road network definition and flow setting to CityFlow, it simulates the dynamics of each vehicle.…”

Section: A Environmentmentioning

confidence: 99%

“…2) It is non-trivial to define a proper reward function that directly optimizes the final target (i.e., average travel time) due to its nature of feedback latency and difficulty on credit assignment. The former is typically solved by building a simulator with high fidelity and training a policy that is well-performed in the simulator [31], [32]. However, the latter problem remains unsolved.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PlanLight: Learning to Optimize Traffic Signal Control With Planning and Iterative Policy Improvement

Zhang

Kafouros

2020

IEEE Access

Self Cite

View full text Add to dashboard Cite

Intelligent traffic signal control (TSC) is essential for transportation efficiency in modern road networks. There is an emerging trend of using deep reinforcement learning techniques to train TSC models in simulators for reducing trial-and-error in real-world scenarios, and recent studies have shown promising results. The target of TSC is to minimize the average travel time of a given area. However, it is impractical to directly optimize the target by setting the average travel time as the reward function due to its nature of feedback latency and difficulty on credit assignment. Existing methods often define the reward function in a heuristic way, which may cause a biased optimization on the real target as they only optimize the accumulative reward. In this work, we propose PlanLight, a novel planning-based TSC algorithm that learns from the demonstration of rollout algorithm, which obtains suboptimal control on the given target, through behavior cloning. We show the effectiveness and efficiency of the rollout algorithm in the multi-intersection control scenario. Moreover, we achieve further policy optimization by improving the base policy in the rollout procedure iteratively. Through comprehensive experiments, we demonstrate that PlanLight outperforms both conventional transportation approaches and existing learning-based methods in various sizes of traffic datasets. Furthermore, we empirically show the potential of PlanLight to be a general algorithm to obtain improvement on future state-of-the-art TSC methods.

show abstract

Section: A Environmentmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

PlanLight: Learning to Optimize Traffic Signal Control With Planning and Iterative Policy Improvement

Zhang

Kafouros

2020

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this paper, we conduct extensive experiments in a new simulator CityFlow [14]. The agent can obtain states of the environment like the number of vehicles through flexible APIs.…”

Section: Experiments 51 Experiments Settingsmentioning

confidence: 99%

Learning Traffic Signal Control from Demonstrations

Xiong

Zheng

et al. 2019

Proceedings of the 28th ACM International Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

Reinforcement learning (RL) has recently become a promising approach in various decision-making tasks. Among them, traffic signal control is the one where RL makes a great breakthrough. However, these methods always suffer from the prominent exploration problem and even fail to converge. To resolve this issue, we make an analogy between agents and humans. Agents can learn from demonstrations generated by traditional traffic signal control methods, in the similar way as people master a skill from expert knowledge. Therefore, we propose DemoLight, for the first time, to leverage demonstrations collected from classic methods to accelerate learning. Based on the state-of-the-art deep RL method Advantage Actor-Critic (A2C), training with demos are carried out for both the actor and the critic and reinforcement learning is followed for further improvement. Results under real-world datasets show that Demo-Light enables a more efficient exploration and outperforms existing baselines with faster convergence and better performance.

show abstract

“…where N i is the neighborhood scope: the set of communication available for target agent, and ι denotes temperature factor. To jointly attend to the neighborhood from different representation subspaces at different grids, we leverage multi-head attention as in previous work [32,33,38,42] to extend the observation as:…”

Section: Multi-head Attention For Coordinationmentioning

confidence: 99%

CoRide

Jin

Zhou

Zhang

et al. 2019

Proceedings of the 28th ACM International Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

How to optimally dispatch orders to vehicles and how to trade off between immediate and future returns are fundamental questions for a typical ride-hailing platform. We model ride-hailing as a large-scale parallel ranking problem and study the joint decisionmaking task of order dispatching and fleet management in online ride-hailing platforms. This task brings unique challenges in the following four aspects. First, to facilitate a huge number of vehicles to act and learn efficiently and robustly, we treat each region cell as an agent and build a multi-agent reinforcement learning framework. Second, to coordinate the agents from different regions to achieve long-term benefits, we leverage the geographical hierarchy of the region grids to perform hierarchical reinforcement learning. Third, to deal with the heterogeneous and variant action space for joint order dispatching and fleet management, we design the action as the ranking weight vector to rank and select the specific order or the fleet management destination in a unified formulation. Fourth, to achieve the multi-scale ride-hailing platform, we conduct the decision-making process in a hierarchical way where a multihead attention mechanism is utilized to incorporate the impacts of neighbor agents and capture the key agent in each scale. The whole novel framework is named as CoRide. Extensive experiments based on multiple cities real-world data as well as analytic synthetic data demonstrate that CoRide provides superior performance in terms of platform revenue and user experience in the task of citywide hybrid order dispatching and fleet management over strong baselines.

show abstract

CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenario

Cited by 167 publications

References 8 publications

PlanLight: Learning to Optimize Traffic Signal Control With Planning and Iterative Policy Improvement

PlanLight: Learning to Optimize Traffic Signal Control With Planning and Iterative Policy Improvement

Learning Traffic Signal Control from Demonstrations

CoRide

Contact Info

Product

Resources

About