Multi-Agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC)

Abstract

The population is steadily increasing worldwide resulting in intractable traffic congestion in dense urban areas. Adaptive Traffic Signal Control (ATSC) has shown strong potential to effectively alleviate urban traffic congestion by adjusting the signal timing plans in real-time in response to traffic fluctuations to achieve the desired objectives (e.g., minimizing delay). Efficient and robust ATSC can be designed using a multi-agent reinforcement learning (MARL) approach in which each controller (agent) is responsible for the control of traffic lights around a single traffic junction. Applying MARL approaches to ATSC problem is associated with a few challenges as agents typically react to changes in the environment at the individual level but the overall behaviour of all agents may not be optimal. This dissertation presents the development and evaluation of a novel system of Multi-Agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC). The MARLIN-ATSC control system is developed to provide a self-learning ATSC using a synergetic combination of reinforcement learning approaches and game theory. MARLIN-ATSC operates in two modes: (1) independent mode, i.e. each intersection controller operates independently of other agents; and (2) integrated mode, where each controller coordinates the signal control actions with the neighbouring intersections. The system was tested on three networks (i.e., small, medium, large-scale) to ensure seamless transferability of the system design and results. The large-scale application was conducted on a computerized testbed network of 60 intersections in the lower downtown core of the City of Toronto for the morning rush hour handling 25,000 trips. The results show unprecedented reduction in the average intersection delay ranging from 27% in mode 1 to 39% in mode 2 at the network level; and travel time savings of 15% in mode 1 and 26% in mode 2, iii along the busiest routes in downtown Toronto. The thesis shows how mathematical modelling of the traffic control problem as a stochastic control problem, combined with the utilisation of artificial intelligence techniques such as reinforcement learning in a game-theory setup, can provide highly useful and economically inexpensive solutions to real-life problems such as urban traffic congestion.

Continuous residual reinforcement learning for traffic signal control optimization

“…In (El-Tantawy et al 2013), a coordinated traffic signal control scheme based on multi-agent reinforcement learning was developed. In this scheme, each agent that controls one intersection coordinates its actions with neighboring intersections.…”

“…Adaptive traffic signal control is a real-time traffic management strategy in which traffic signal timing changes, or adapts, according to the actual traffic demand. It uses the observed information to immediately adapt to traffic demand (Aslani et al 2017, El-Tantawy et al 2013.…”

Routing Algorithm and Traffic Light Control based on Vehicular Delay-Tolerant Networks

“…For special application scenarios, Leu et al [21] propose an intelligent traffic light control system for an ambulance to find a neighboring hospital quickly. El-Tantawy et al [22] describe two different adaptive traffic light control modes: independent mode and cooperative mode. In independent mode, each control module works independently in an intersection, and in collaborative mode, each controller collaborates with the adjacent one.…”

Context-aware multi-modal traffic management in ITS: A Q-learning based algorithm

“…Although the use of several approaches to investigate traffic management issues, the majority of these propositions are based on the use of reinforcement and Qlearning techniques [7][8][9][10][11][12][13][14][15][16]. Among those works, the timearrival estimation technique introduced in [7] proposed a prediction engine system that built its visions and decisions based on the context behaviors of drivers and vehicles.…”