This paper develops an adaptive traffic signal control method that optimizes performance over time. The adaptive controller, instead of retrieving control parameters from a library according to changes in traffic, allows the value of control parameters to be slowly time-varying so that controller's behavior conforms to new circumstance. The adaptive controller is developed on Approximate Dynamic Programming (ADP). By replacing the look-up table containing real value of each state combination with an approximate function, ADP obviates the limit of Dynamic programming in cases of high dimensionality and of incomplete information. Approximation is progressively improved by employing online learning techniques. In this paper, we investigate ADP approach with reinforcement learning and perturbation learning. Through numerical examples, this paper shows that an adaptive controller based on ADP method is operational at isolated intersections while offering potentials for network control. It also shows that a simple linear approximation is sufficient to provide practical significance and to improve performance substantially towards theoretical limit. ADP is a theoretical framework that allows various approximation and learning techniques to be employed to overcome the restrictions of Dynamic Programming (DP) in real-time adaptive control. Dynamic programming, developed by Richard Bellman, traditionally, is the only exact and efficient way to solve problems in optimization over time, in the general case where noise and nonlinearity are present (Werbos, 2004). Key to the problem solving process in DP is the backward recursive estimation of the value of being in a state of the current stage. Subject to the probability distribution of exogenous information, an optimal decision is to optimize the overall utility in the process of transferring the state of current stage to the next state of the following stage. In doing so, DP guarantees global optimum in problem solving. Powerful as it is, DP's usefulness in real-time control is severely limited. This is because of the exponential relationship between computational burden and the size of state space, the complexity of probability distribution of information, and the number of optional decisions. This relationship is often referred as 'Three Curses of Dimensionality' (Powell, 2007). Yet, this is not the end. Supplying complete information for evaluating policies, as required by DP, is unrealistic in real-time control. In the introduction to fundamentals of ADP, we will show that, using an approximate function to replace the exact values in DP, ADP drastically reduces computational burden, and by adopting a forward algorithmic strategy, ADP utilizes limited real-time information to evaluate control policy and update approximation. The reason of introducing reinforcement learning is that parameters of the approximate function are not known a priori, and we need a technique to learn the proper values in a changing environment where a 'teacher' is not available. Such a learning pr...
