Deep Reinforcement Learning Agent with Varying Actions Strategy for Solving the Eco-Approach and Departure Problem at Signalized Intersections

Abstract

<jats:p> Eco-approach and departure is a complex control problem wherein a driver’s actions are guided over a period of time or distance so as to optimize fuel consumption. Reinforcement learning (RL) is a machine learning paradigm that mimics human learning behavior, in which an agent attempts to solve a given control problem by interacting with the environment and developing an optimal policy. Unlike the methods implemented in previous studies for solving the eco-driving problem, RL does not require prior knowledge of the environment to be learned and processed. This paper develops a deep reinforcement learning (DRL) agent for solving the eco-approach and departure problem in the vicinity of signalized intersections for minimization of fuel consumption. The DRL algorithm utilizes a deep neural network for the RL. Novel strategies such as varying actions, prioritized experience replay, target network, and double learning were implemented to overcome the expected instabilities during the training process. The results revealed the significance of the DRL algorithm in reducing fuel consumption. Interestingly, the DRL algorithm was able to successfully learn the environment and guide vehicles through the intersection without red light running violation. On average, the DRL provided fuel savings of about 13.02% with no red light running violations. </jats:p>

Intersection control with connected and automated vehicles: a review

“…Shi et al (2018) used traditional (non-deep) Q-learning to develop an efficient driving strategy for approaching signalized intersections. Mousa et al (2020) used deep Q-learning with prioritized experience replay, target networks and double-learning to train an RL agent to approach and depart efficiently at signalized intersections for situations where no other vehicles are interfering. Wang et al (2022a, 2022b) focused on the CAV control problem in mixed traffic flow at signalized intersections with particular considerations of the oscillations induced by human drivers.…”

Eco-driving for Electric Connected Vehicles at Signalized Intersections: A Parameterized Reinforcement Learning approach

“…Controlling of CV is one representative in this field, as the vehicle can move longitudinally and laterally on the road. While the majority of the existing studies are confined to car following motion (Shi et al, 2018;Mousa et al, 2020;Zhou et al, 2020;Wegener et al, 2021), this paper develops a parameterized action space to naturally describe the control problem with hybrid actions and thereby implement joint optimization of car-following and lane-changing movement. Figure 3 presents an example of parameterized action space.…”

“…As one of the value-based reinforcement learning algorithms, the Q-learning approach cannot control the vehicle in continuous acceleration space, and thereby causes local optimum and uneven trajectory in most cases. The framework developed by Mousa et al (2020) provided insight into the DRL-based eco-driving system, which introduced deep Q network (DQN) to improve the fuel performance of the controlled CV. However, major disadvantage similar to the work of Shi et al (2018) was encountered in their study, namely, the losing efficacy in continuous action space.…”

Navigating Electric Vehicles Along a Signalized Corridor via Reinforcement Learning: Toward Adaptive Eco-Driving Control

“…It is usually accomplished by maintaining a smooth driving speed and avoiding sudden acceleration as much as possible. The eco-driving concept is conventionally studied for gasoline vehicles ( 3–7 ) with the intent to attain fuel-saving and emission-reducing goals. However, with the development and deployment of the prevailing electric vehicles (EVs), relative studies were also developed to achieve eco-driving for EVs ( 8, 9 ) and mixed gasoline and electric vehicles ( 10 ).…”

“…Meanwhile, the action space cannot be very large as they also applied the value-based RL algorithm. Similarly, the method given by Mousa et al still faces the defects of discretized action ( 5 ). The deep deterministic policy gradient (DDPG) approach is widely used to address the above issue with stable performance ( 27 ).…”