Application of a Deep Deterministic Policy Gradient Algorithm for Energy-Aimed Timetable Rescheduling Problem

Abstract

Reinforcement learning has potential in the area of intelligent transportation due to its generality and real-time feature. The Q-learning algorithm, which is an early proposed algorithm, has its own merits to solve the train timetable rescheduling (TTR) problem. However, it has shortage in two aspects: Dimensional limits of action and a slow convergence rate. In this paper, a deep deterministic policy gradient (DDPG) algorithm is applied to solve the energy-aimed train timetable rescheduling (ETTR) problem. This algorithm belongs to reinforcement learning, which fulfills real-time requirements of the ETTR problem, and has adaptability on random disturbances. Superior to the Q-learning, DDPG has a continuous state space and action space. After enough training, the learning agent based on DDPG takes proper action by adjusting the cruising speed and the dwelling time continuously for each train in a metro network when random disturbances happen. Although training needs an iteration for thousands of episodes, the policy decision during each testing episode takes a very short time. Models for the metro network, based on a real case of the Shanghai Metro Line 1, are established as a training and testing environment. To validate the energy-saving effect and the real-time feature of the proposed algorithm, four experiments are designed and conducted. Compared with the no action strategy, results show that the proposed algorithm has real-time performance, and saves a significant percentage of energy under random disturbances.

A Real-Time Train Timetable Rescheduling Method Based on Deep Learning for Metro Systems Energy Optimization under Random Disturbances

“…But the value of the disturbance is random. e time model defines the departure instant t de , travel time t tr , and dwell time t dw of each train at each station [33]. e starting station is defined as station no.…”

“…If the braking power is less than the traction power, it can be fully used; otherwise, resistors will kick in and consume the overflowing braking power to maintain the train voltage under a safe value. e minimum value of traction power and braking conversion power is defined as P F [33]:…”

A Hybrid End-to-End Control Strategy Combining Dueling Deep Q-network and PID for Transient Boost Control of a Diesel Engine with Variable Geometry Turbocharger and Cooled EGR

“…Reinforcement learning is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with environment (see Figure 4) [28,29]. For every state s, the agents always try to maximize the expected discounted return by choosing an action a.…”

Boundary Identification for Traction Energy Conservation Capability of Urban Rail Timetables: A Case Study of the Beijing Batong Line

“…Nitisiri et al [29] developed a parallel multi-objective evolutionary algorithm with hybrid sampling and learning-based mutation to solve the train scheduling problem. To further investigate the rail timetable rescheduling problem, Yang et al [30] introduced the Deep Deterministic Policy Gradient (DDPG) algorithm over a conventional Q-learning strategy to realize the energy-aimed rescheduling by adjusting the cruising speed and dwelling time continuously. It can be found that energy consumption level of a given timetable is actually an inherent attribute.…”