Robust Deep Reinforcement Learning for Traffic Signal Control

Tan, Kai Liang; Sharma, Anuj; Sarkar, Soumik

doi:10.1007/s42421-020-00029-6

Cited by 16 publications

(7 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deep reinforcement learning methods have demonstrated robustness to sensor failures (Rodrigues and Azevedo, 2019;Tan et al, 2020). Furthermore, by using the transfer learning technique (Tan et al, 2020), the trained model can also handle demand surges. However, the above methods do not adapt to new road networks.…”

Section: E Summary Of Previous Work On Robustness and Generalizabilit...mentioning

confidence: 99%

Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control

Shi,

Devailly,

Larocque

et al. 2024

IEEE Open J. Intell. Transp. Syst.

View full text Add to dashboard Cite

A number of deep reinforcement-learning (RL) approaches propose to control traffic signals.Compared to traditional approaches, RL approaches can learn from higher-dimensionality input road and vehicle sensors and better adapt to varying traffic conditions resulting in reduced travel times (in simulation). However, these RL methods require training from massive traffic sensor data. To offset this relative inefficiency, some recent RL methods have the ability to first learn from small-scale networks and then generalize to unseen city-scale networks without additional retraining (zero-shot transfer). In this work, we study the robustness of such methods along two axes. First, sensor failures and GPS occlusions create missing-data challenges and we show that recent methods remain brittle in the face of these missing data. Second, we provide a more systematic study of the generalization ability of RL methods to new networks with different traffic regimes. Again, we identify the limitations of recent approaches. We then propose using a combination of distributional and vanilla reinforcement learning through a policy ensemble. Building upon the state-of-the-art previous model which uses a decentralized approach for large-scale traffic signal control with graph convolutional networks (GCNs), we first learn models using a distributional reinforcement learning (DisRL) approach. In particular, we use implicit quantile networks (IQN) to model the state-action return distribution with quantile regression. For traffic signal control problems, an ensemble of standard RL and DisRL yields superior performance across different scenarios, including different levels of missing sensor data and traffic flow patterns. Furthermore, the learning scheme of the resulting model can improve zero-shot transferability to different road network structures, including both synthetic networks and real-world networks (e.g., Luxembourg, Manhattan). We conduct extensive experiments to compare our approach to multi-agent reinforcement learning and traditional transportation approaches. Results show that the proposed method improves robustness and generalizability in the face of missing data, varying road networks, and traffic flows.

show abstract

Section: E Summary Of Previous Work On Robustness and Generalizabilit...mentioning

confidence: 99%

Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control

Shi,

Devailly,

Larocque

et al. 2024

IEEE Open J. Intell. Transp. Syst.

View full text Add to dashboard Cite

show abstract

“…They evaluated their algorithm with a simulation of probabilistic detector failure. As is done in adversarial machine learning, [71] injected Gaussian noise into queue length observations, and validated their approach with simulations where trucks cause overestimated vehicle counts. Meanwhile, to handle miscalibrated measurements, [49] combined next state prediction with imitation learning from a real traffic controller (SCOOTS).…”

Section: Progress Toward Solutionsmentioning

confidence: 99%

The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Reality

Chen¹,

Fang²,

Sadeh³

2022

Preprint

View full text Add to dashboard Cite

Traffic signal control (TSC) is a high-stakes domain that is growing in importance as traffic volume grows globally. An increasing number of works are applying reinforcement learning (RL) to TSC; RL can draw on an abundance of traffic data to improve signalling efficiency. However, RL-based signal controllers have never been deployed. In this work, we provide the first review of challenges that must be addressed before RL can be deployed for TSC. We focus on four challenges involving (1) uncertainty in detection, (2) reliability of communications, (3) compliance and interpretability, and (4) heterogeneous road users. We show that the literature on RL-based TSC has made some progress towards addressing each challenge. However, more work should take a systems thinking approach that considers the impacts of other pipeline components on RL.

show abstract

“…Hereafter, researchers developed several methods and techniques, including saturation flow ratio method (44) and regression technique (45), to obtain static or dynamic PCE of the truck to consider freight traffic in signal control. Recently, freight traffic has been treated as noise or perturbation into statespace to examine the robustness of Reinforcement Learning methods on multi-modal traffic signal control (46).…”

Section: Literature Reviewmentioning

confidence: 99%

Lyapunov Function Consistent Adaptive Network Signal Control with Back Pressure and Reinforcement Learning

Ma¹,

Wang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

This research studies the network traffic signal control problem. It uses the Lyapunov control function to derive the back pressure method, which is equal to differential queue lengths weighted by intersection lane flows. Lyapunov control theory is a platform that unifies several current theories for intersection signal control. We further use the theorem to derive the flow-based and other pressure-based signal control algorithms. For example, the Dynamic, Optimal, Real-time Algorithm for Signals (DORAS) algorithm may be derived by defining the Lyapunov function as the sum of queue length. The study then utilizes the back pressure as a reward in the reinforcement learning (RL) based network signal control, whose agent is trained with double Deep Q-Network (Double-DQN). The proposed algorithm is compared with several traditional and RL-based methods under passenger traffic flow and mixed flow with freight traffic, respectively. The numerical tests are conducted on a single corridor and on a local grid network under three traffic demand scenarios of low, medium, and heavy traffic, respectively. The numerical simulation demonstrates that the proposed algorithm outperforms the others in terms of the average vehicle waiting time on the network.

show abstract

Robust Deep Reinforcement Learning for Traffic Signal Control

Cited by 16 publications

References 25 publications

Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control

Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control

The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Reality

Lyapunov Function Consistent Adaptive Network Signal Control with Back Pressure and Reinforcement Learning

Contact Info

Product

Resources

About