Ramp Metering Control Based on the Q-Learning Algorithm

Ivanjko, Edouard; Nečoska, Daniela Koltovska; Gregurić, Martin; Vujić, Miroslav; Jurković, Goran; Mandžuka, Sadko

doi:10.1515/cait-2015-0019

Cited by 10 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the view of certain limitations to ramp metering, Meshkat (2015) et al applied a prioritized control strategy to distribute traffic flow more rationally across the entire road network [ 23 ]. For the current difficulties in controlling discrete partial differential equation systems, Belletti (2017) et al designed a new multi-intelligent parametric-free control algorithm, and showed how to realize parametric-free control based on BP neural network [ 24 – 27 ]. In addition, a more accurate BeATS simulator is adopted by researchers, which is the most advanced parameter control system compared with ALINEA, and achieved comparable control effect.…”

Section: Literature Reviewmentioning

confidence: 99%

Freeway ramp metering based on PSO-PID control

Wei

Long

et al. 2021

PLoS ONE

View full text Add to dashboard Cite

Ramp metering on freeway is one of the effective methods to alleviate traffic congestion. This paper advances the field of freeway ramp metering by introducing an application to the on-ramp, capitalizing on the macro traffic follow theory and improved the freeway traffic flow. The Particle Swarm Optimization (PSO) based on Proportional Integral Derivative (PID) controller is further developed to single ramp metering as well as to optimize the PID parameters. The approach is applied to a case study of the Changyi Freeway(G5513) in Hunan, China. The simulation is conducted by applying the actual profile traffic data to PID controller to adjust the entering traffic flow on the freeway on-ramp. The results show that the PSO-PID controller tends to converge in about 80 minutes, and the density tends to be stable after 240 iterations. The system has smaller oscillation, more accurate adjustment of ramp regulation rate, and more ideal expected traffic flow density. The traffic congestion on mainline is effectively slowed down, traffic efficiency is improved, and travel time and cost are reduced. The nonlinear processing ability of PSO-PID controller overcomes the defects of the traditional manual closing ramp, and can be successfully applied in the field of intelligent ramp metering.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Freeway ramp metering based on PSO-PID control

Wei

Long

et al. 2021

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…By the type of the applied Q-learning method, these studies can be classified into two categories. e first category consists of those that used lookup table methods, i.e., [17,18,[20][21][22][23][24][25][26][27][28][29][30][31]; the second category includes those that employed value function approximation-based methods, i.e., [31][32][33].…”

Section: Q-learning Applications In Freeway Controlmentioning

confidence: 99%

“…Lookup table methods, also known as tabular methods [34], as suggested by the name, maintain a lookup [17,18,[20][21][22][23]. Some other studies employed more sophisticated methods, e.g., k-nearest neighbors, to approximate continuous state spaces.…”

Section: Q-learning Applications In Freeway Controlmentioning

confidence: 99%

“…Hence, simulation models play an important role in reinforcement learning. Among the studies summarized in this section, [19,22,24,30,33] employed microscopic traffic simulation models such as VISSIM, Paramics, and SUMO; [17,18,20,21,23,31,32] used macroscopic dynamic traffic flow models such as CTM [16] and METANET [14] as the simulation tools.…”

Section: Q-learning Applications In Freeway Controlmentioning

confidence: 99%

See 1 more Smart Citation

Ramp Metering for a Distant Downstream Bottleneck Using Reinforcement Learning with Value Function Approximation

Zhou

Özbay

Kachroo

et al. 2020

Journal of Advanced Transportation

View full text Add to dashboard Cite

Ramp metering for a bottleneck located far downstream of the ramp is more challenging than for a bottleneck that is near the ramp. This is because under the control of a conventional linear feedback-type ramp metering strategy, when metered traffic from the ramp arrive at the distant downstream bottleneck, the state of the bottleneck may have significantly changed from when it is sampled for computing the metering rate; due to the considerable time, these traffic will have to take to traverse the long distance between the ramp and the bottleneck. As a result of such time-delay effects, significant stability issue can arise. Previous studies have mainly resorted to compensating for the time-delay effects by incorporating predictors of traffic flow evolution into the control systems. This paper presents an alternative approach. The problem of ramp metering for a distant downstream bottleneck is formulated as a Q-learning problem, in which an intelligent ramp meter agent learns a nonlinear optimal ramp metering policy such that the capacity of the distant downstream bottleneck can be fully utilized, but not to be exceeded to cause congestion. The learned policy is in pure feedback form in that only the current state of the environment is needed to determine the optimal metering rate for the current time. No prediction is needed, as anticipation of traffic flow evolution has been instilled into the nonlinear feedback policy via learning. To deal with the intimidating computational cost associated with the multidimensional continuous state space, the value function of actions is approximated by an artificial neural network, rather than a lookup table. The mechanism and development of the approximate value function and how learning of its parameters is integrated into the Q-learning process are well explained. Through experiments, the learned ramp metering policy has demonstrated effectiveness and benign stability and some level of robustness to demand uncertainties.

show abstract

“…Liang et al [16] designed a traffic density controller based on LWR (Lighthill-Whitham-Richards) model and RBF neural network, which could keep the freeway traffic at a setting traffic density. Ivanjko et al [17] proposed a Q-learning-based ramp control algorithm using downstream speed and ramp queuing length as the state space. van de Weg et al [18] combined ALINEA with variable speed limit strategy to optimize the upstream and downstream speed boundaries and related parameters in ALINEA.…”

Section: Introductionmentioning

confidence: 99%

A Ramp Metering Method Based on Congestion Status in the Urban Freeway

Liu

Cao

et al. 2020

IEEE Access

View full text Add to dashboard Cite

In ramp metering methods, the ALINEA algorithm is a very effective way and has been applied widely. But the critical occupancy in ALINEA algorithm is often difficult to obtain and not particularly accurate. It will greatly affect the performance of ALINEA algorithm. In this paper, an improved ALINEA algorithm, named CS-ALINEA, is proposed. In this method, the traffic flow is used to replace the occupancy as the control parameter and the control rate can be selected according to the congestion status reclassified adaptively. In the existing ramp control methods, to guarantee the traffic efficiency of mainstream, the impact of ramp overflow on ground road traffic is often ignored. In order to resolve this issue, the segmented control method is adopted in this paper. When the ramp queuing length exceeds the critical queue length, the signal timing scheme is adjusted by selecting the control rate to avoid the ramp overflow. The SUMO simulation platform is used to simulate the ramp control and test the CS-ALINEA algorithm. The experimental results show that the proposed method can optimize the ramp queuing length and reduce waiting time of vehicles while the efficiency of urban freeway can be guaranteed.

show abstract

Ramp Metering Control Based on the Q-Learning Algorithm

Cited by 10 publications

References 7 publications

Freeway ramp metering based on PSO-PID control

Freeway ramp metering based on PSO-PID control

Ramp Metering for a Distant Downstream Bottleneck Using Reinforcement Learning with Value Function Approximation

A Ramp Metering Method Based on Congestion Status in the Urban Freeway

Contact Info

Product

Resources

About