An automated signalized junction controller that learns strategies by temporal difference reinforcement learning

Abstract

This paper shows how temporal difference learning can be used to build a signalized junction controller that will learn its own strategies though experience. Simulation tests detailed here show that the learned strategies can have high performance. This work builds upon previous work where a neural network based junction controller that can learn strategies from a human expert was developed . In the simulations presented, vehicles are assumed to be broadcasting their position over WiFi giving the junction controller rich information. The vehicle's position data are pre-processed to describe a simplified state. The state-space is classified into regions associated with junction control decisions using a neural network. This classification is the strategy and is parametrized by the weights of the neural network. The weights can be learned either through supervised learning with a human trainer or reinforcement learning by temporal difference (TD). Tests on a model of an isolated T junction show an average delay of 14.12 s and 14.36 s respectively for the human trained and TD trained networks. Tests on a model of a pair of closely spaced junctions show 17.44 s and 20.82 s respectively. Both methods of training produced strategies that were approximately equivalent in their equitable treatment of vehicles, defined here as the variance over the journey time distributions.

Supervised learning from human performance at the computationally hard problem of optimal traffic signal control on a network of junctions

“…a neural network. This has been demonstrated by Tesauro [ 38 ] in application to computer backgammon programs, and more recently by the author [ 22 ] in application to traffic signal control.…”

“…The MOVA [ 20 ] and the SCOOT [ 21 ] systems are in wide use today [ 31 ] and use inductive loop measurements to inform signal control decisions. TD control [ 22 ] is applied using measurements of individual vehicle's position and speed—albeit in a compressed form. This is to provide an example of how approximate optimization methods may perform with high quality traffic state data.…”

“…The TD control system as described in [ 22 ] was employed. As in [ 22 ], a compressed form of individual vehicle position and speed data are used to describe the control state space. This state space is described in detail in the context of HuTMaC in §5.2.2.…”

“…This paper presents a microsimulation-based evaluation of a number of traffic-responsive signal control systems. As benchmark controllers, we employ two systems that are widely deployed in cities today: the MOVA system [ 20 ] and the SCOOT system [ 21 ]; as well as a temporal difference (TD) learning-based system, developed in part by the author [ 22 ].…”

Network Traffic Control in Cyber-Transportation Systems Accounting for User-Level Fairness

“…Researchers also focus on using machine-learning techniques in the CV environment to improve signal settings (Box et al, 2011;Box et al, 2012;and Box & Waterson, 2013).…”