Reinforcement Learning in Urban Network Traffic Signal Control: A Systematic Literature Review

Abstract

Improvement of traffic signal control (TSC) efficiency has been found to lead to improved urban transportation and enhanced quality of life. Recently, the use of reinforcement learning (RL) in various areas of TSC has gained significant traction; thus, we conducted a systematic literature review as a systematic, comprehensive, and reproducible review to dissect all the existing research that applied RL in the network-level TSC (NTSC) domain. The review only targeted the network-level articles that tested the proposed methods in networks with two or more intersections. We used natural language processing to define the search strings and searched Google Scholar, Web of Science, IEEE Xplore, ACM Digital Library, Springer Link, and Science Direct databases. This review covers 160 peer-reviewed articles from 30 countries published from 1994 to March 2020. The goal of this study is to provide the research community with statistical and conceptual knowledge, summarize existence evidence, characterize RL applications in NTSC domains, explore all applied methods and major first events in the defined scope, and identify areas for further research based on the explored research problems in current research.

The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Reality

“…Likewise, [33] reviewed problem formulations and the history of algorithmic developments for RL-based TSC as part of a wider review on deep RL for intelligent traffic systems. Finally, [10] performed a highly systematic overview of the past 26 years of research in this domain that provides quantitative support for some of the patterns that we identify.…”

“…Every approach (roadway entering the intersection) is split into lanes for forward and left-turn movements, with right turns assumed to always be permissible [12]. For efficiency, pairs of compatible movements are typically arranged into phases and signalled simultaneously [10,13]. Then, the task is to find some division of green time between phases for each intersection in a road network, which maximizes metrics such as the throughput of the road network.…”

“…We refer the reader to [14] for details of the problem formulation. Different approaches to this division of green time exist, including choosing phase durations or phase sequences, or fixing a phase sequence within a cycle and choosing the length of the cycle or the proportions of each phase within the cycle [10,13]. Three main types of algorithmic approaches exist.…”

“…For reasons of tractability, 𝑠 𝑡 is usually some kind of abstracted, numerical representation of the intersection. As reviewed by [10], five of the most commonly included elements of state representations at the intersection level are (1) the numbers of queueing vehicles in lanes (queue length, 38%), (2) the current phase (11%), (3) the number of vehicles (10%), (4) the positions of vehicles (6%), and (5) the speeds of vehicles (6%).…”

“…However, to our knowledge, RL-based TSC algorithms have never been deployed. This is in spite of the fact that papers introducing novel algorithms in this area commonly list real-world deployment as a goal for future work [10]. We believe that this discrepancy has arisen due to a focus on methodological contributions, over a holistic systems thinking approach based on the data-to-deployment pipeline [11].…”