Time Difference Penalized Traffic Signal Timing by LSTM Q-Network to Balance Safety and Capacity at Intersections

Abstract

The conflict between limited road resources and rapid car ownership makes the traffic signal timing become a pivotal challenge. Emerging studies have been carried out on adaptive signal timing, but most of them still focus on the throughput of intersections, leaving safety and travel experience unconsidered. This paper proposes a time difference penalized traffic signal timing method by reinforcement learning technique to balance safety and throughput capacity in traffic control system. Firstly, a microcosmic state representation is proposed to integrate the dynamics of both traffic lights and road vehicles, including driver behaviors of lane changing, car-following, previous phase of traffic light and its duration. Secondly, an action space, including 8 signal phases, and a behavior-aware reward function are designed to resist the red-light overflow. Finally, a partial long short term memory (LSTM) network is trained to balance traffic efficiency and traveling experience. In the network training, a parallel sampling method is adopted to obtain experience from multiple environments to accelerate the training convergence in practical application. Experimental results show that the proposed method improves the intersection efficiency up to 14.28% compared to the fixed signal timing and 5.26% compared to DQN while getting rid of red-light overflow time.

The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Reality

“…Similarly, [112] used a binary logistic crash risk model to define crash penalties while also minimizing waiting time. Using a state formulation based on individual signals, [113] regularized the red light duration of signals to mitigate unsafe behaviour caused by driver frustration with extended red lights. [114] included yellow change intervals in their action space and added a penalty for emergency braking by vehicles.…”

Spatio-Temporal Wind Speed Prediction Based on Variational Mode Decomposition

“…LSTM is proposed as the main component of BiLSTM to overcome limitations such as gradient vanishing and long-term dependence of recurrent neural networks (RNN). Furthermore, because of its superiority in dealing with sequential data, LSTM has found widespread application in video analysis [24], speech recognition [25], signal analysis [26], etc.…”

Adaptive Optimization of Traffic Signal Timing via Deep Reinforcement Learning

“…For the formulation of rewards, Liang et al [21] proposed using the cumulative vehicle's waiting time difference at the intersection before and after the traffic light action as the reward equation. Liao et al [27] put forward the corresponding penalty items when setting the reward equation in order to avoid the excessively long green time which will cause traffic loss in all directions at the intersection. Combining the above viewpoints, the definition of reward equation in this study will be measured from two dimensions.…”