A Distributed Control Method for Urban Networks Using Multi-Agent Reinforcement Learning Based on Regional Mixed Strategy Nash-Equilibrium

Abstract

Urban network traffic congestion can be caused by disturbances, such as fluctuation and disequilibrium of traffic demand. This paper designs a distributed control method for preventing disturbancebased urban network traffic congestion by integrating Multi-Agent Reinforcement Learning (MARL) and regional Mixed Strategy Nash-Equilibrium (MSNE). To enhance the disturbance-rejection performance of Urban Network Traffic Control (UNTC), a regional MSNE concept is integrated, which models the competitive relationship between each agent and its neighboring agents in order to improve the decisionmaking process of MARL. The learning rate is enhanced with a self-adaptive ability to avoid a local optimal dilemma; Jensen-Shannon (JS) divergence is utilized to define the learning rate of the modified MARL. A two-way rectangular grid network with nine intersections is modeled via a Cell Transmission Model (CTM). A probability distribution mechanism, which can update the turn ratio of each approach dynamically and discretely, is established to represent the segmented route-decision process of the vehicles. The effectiveness of the proposed control method is evaluated through simulations in the grid network. The results show the influence of major disturbances, such as fluctuation of vehicle arrival rate, fluctuation of traffic demand (e.g. a rapidly rising flow and extreme changes in origin-destination distribution), and disequilibrium of traffic demand (e.g. different arrival flows at each boundary of the urban network), on the performance of the suggested control method. The results can be used to improve the state of the art in order to reduce urban network traffic congestion due to these disturbances. INDEX TERMS Urban network traffic control, distributed traffic signal control system, multi-agent reinforcement learning, mixed strategy Nash-equilibrium, numerical simulation.

Reinforcement Learning in Urban Network Traffic Signal Control: A Systematic Literature Review

“…(El-Tantawy and Abdulhai, 2010;El-Tantawy et al, 2013). Famous methods proposed in traffic theory context, such as CTM (Chanloha et al, 2014;Ajorlou et al, 2015;Qu et al, 2020), Max-plus (Kuyer et al, 2008;Medina and Benekohal, 2012), and Max Pressure (MP) or back pressure (brought in NTSC from the communications networks theory) (Wei et al, 2019a), are also applied in some research, and in other studies multiple traffic optimization goals are simultaneously optimized (multi-policy RL), e.g. (Dusparic et al, 2016).…”

A Distributed Assignment Method for Dynamic Traffic Assignment Using Heterogeneous-Adviser Based Multi-Agent Reinforcement Learning

“…The state of each agent is the vector of states at all links (edges) in TN. Besides, it is worth noting that rationalize the size of state space to reduce computational complexity [30]. Fortunately, the state of TN only needs to be detected once to be accessible by all agents.…”

“…It is necessary to define a special reward function modality for main agents. .Formula (27) shows that the travel cost input variable of a sub-action has two components: subaction effect estimation † j defined by integrating equations (29) and (30). Formula (28) models the utility among (28) and (29) can be defined in a similar way to (17).…”

“…Formula (28) models the utility among (28) and (29) can be defined in a similar way to (17). The weight of external utility assigned to sub-action † j i a is defined in Formula (30). Similar to formula (18), formula (31) describes a coefficient † j  to ensure the plus or minus characteristic of formula (31).…”

“…Recently, Multi-Agent Systems (MAS) and Reinforcement Learning (RL) have been integrated and applied to the field of traffic management, such as traffic control [29]- [30] and route planning [31]- [32]. With the advantages of both MAS and RL, Multi-Agent Reinforcement Learning (MARL) was introduced for TA [33].…”