A least squares temporal difference actor–critic algorithm with applications to warehouse management

Abstract

This paper develops a new approximate dynamic programming algorithm for Markov decision problems and applies it to a vehicle dispatching problem arising in warehouse management. The algorithm is of the actor-critic type and uses a least squares temporal difference learning method. It operates on a sample-path of the system and optimizes the policy within a prespecified class parameterized by a parsimonious set of parameters. The method is applicable to a partially observable Markov decision process setting where the measurements of state variables are potentially corrupted and the cost is only observed through the imperfect state observations. We show that under reasonable assumptions, the algorithm converges to a locally optimal parameter set. We also show that the imperfect cost observations do not affect the policy and the algorithm minimizes the true expected cost.In the warehouse application, the problem is to dispatch sensor-equipped forklifts in order to minimize operating costs involving product movement delays and forklift maintenance. We consider instances where standard dynamic programming is computationally intractable. Simulation results confirm the theoretical claims of the paper and show that our algorithm converges more smoothly than earlier actor-critic algorithms while substantially outperforming heuristics used in practice.

Learning from animals: How to Navigate Complex Terrains

“…Hence, it can not be guaranteed to obtain a global optimal solution. Convergence results [22,23] establish that it converges to a neighborhood of a stationary point of the expected average reward with probability one (w.p.1).…”

“…One approach to solve this problem is to use an actor-critic algorithm [21]. This paper uses a modified version of a Least-Squares Temporal Difference (LSTD) actor-critic algorithm developed in [22].…”

Temporal logic motion control using actor–critic methods

“…As a property of inner products, the linear approximation in (6) does not change the estimate of the gradient ∇ α ¯ ( θ ) if the optimal coefficient r ★ in (7) is used. Furthermore, the linear approximation reduces the complexity of learning from the space R | X | | U | to the space R n , where n is the dimensionality of θ (Konda, 2002; Estanjini et al, 2012).…”

“…Algorithm 1 learns the critic parameters using a LSTD method, which has been shown to be superior to other stochastic learning methods in terms of the convergence rate (Konda and Tsitsiklis, 2003; Boyan, 1999). Estanjini et al (2012) proposed and established the convergence of a LSTD actor–critic method similar to Algorithm 1 for problems of minimizing expected average costs . In comparison, the goal of the Problem 3.10 in this paper is to minimize an expected total cost (cf.…”

“…By approximating both the policy and state–action value function with a parameterized structure, actor–critic algorithms use much less memory compared with other dynamic programming techniques. In particular, actor–critic algorithms with least squares temporal difference (LSTD) learning have been shown recently to be a powerful tool for large-sized problems (Konda and Tsitsiklis, 2003; Estanjini et al, 2011, 2012). Ding et al (2011) showed that a motion control problem with temporal logic specifications could be converted to a maximal reachability probability (MRP) problem, i.e.…”