Lambda‐Policy Iteration: A Review and a New Implementation

Abstract

In this paper we discuss λ-policy iteration, a method for exact and approximate dynamic programming. It is intermediate between the classical value iteration (VI) and policy iteration (PI) methods, and it is closely related to optimistic (also known as modified) PI, whereby each policy evaluation is done approximately, using a finite number of VI. We review the theory of the method and associated questions of bias and exploration arising in simulation-based cost function approximation. We then discuss various implementations, which offer advantages over well-established PI methods that use LSPE(λ), LSTD(λ), or TD(λ) for policy evaluation with cost function approximation. One of these implementations is based on a new simulation scheme, called geometric sampling, which uses multiple short trajectories rather than a single infinitely long trajectory. INTRODUCTIONApproximate dynamic programming (DP for short) has attracted substantial research interest, and has a wide range of applications, because of its potential to address large and complex problems that may not be treatable in other ways. The literature on the subject is very extensive, and includes several textbooks, research monographs, and surveys that relate to the computational context of this paper. For a nonexhaustive list, we mention the books by Bertsekas and Tsitsiklis [BeT96], Sutton and Barto [SuB98], Gosavi [Gos03], Cao [Cao07], Chang, Fu, Hu, and Marcus [CFH07], Meyn [Mey07], Powell [Pow07], Borkar [Bor08], Haykin [Hay08], Busoniu, Babuska, De Schutter, and Ernst [BBD10], and the author's text in preparation [Ber11a]; the edited volumes and special issues by White and Sofge [WhS92], Si, Barto, Powell, and Wunsch [SBP04], Lewis, Lendaris, and Liu [LLL08], and the 2007-2009 Proceedings of the IEEE Symposium on Approximate Dynamic Programming and Reinforcement Learning; and the recent surveys by Borkar [Bor09], Lewis and † are due to Bruno Scherrer for helpful comments, to Huizhen Yu for related collaboration, and to Mengdi Wang for assistance with computational experimentation. Vrabie [LeV09], Werbos [Wer09], Szepesvari [Sze10], and Bertsekas [Ber11b].The purpose of this paper is to critically review and extend a class of methods for exact and approximate DP, which are based on the λ-policy iteration (λ-PI) method, proposed by Bertsekas and Ioffe [BeI96]. This method is intermediate between the classical value iteration (VI) and policy iteration (PI) methods, and it is closely related to optimistic (also known as modified) PI, whereby each policy evaluation is done approximately, using a finite number of VI. It was originally used as the starting point for the development of approximate simulation-based DP methods of the temporal difference (TD) type, such as LSPE(λ) (see [BeI96], and also [BeT96], Sections 2.3.1 and 8.3). The emphasis in this paper is on implementations of λ-PI, which provide alternatives to approximate PI methods that use other more established methods for policy evaluation.We will focus on the α-discounted n-state Markovian Decisi...

Constrained Optimal Tracking Control of Unknown Systems: A Multi-Step Linear Programming Approach

“…Corollary 1: Let {Q i (z, a)} generated by ( 10) and ( 11), and {Q i V I (z, a)} the sequence generated by the standard VI algorithm corresponding to setting H i = 1 for all i in (11) . Under assumption (12)…”

“…1 All authors are with the Department of Information Technology and Electrical Engineering, ETH Zurich, Switzerland, {atanzana,jlygeros}@ethz.ch presented in [11]. A class of PI algorithms based on temporal difference learning and the λ-operator is proposed in [12], which has been further extended using abstract dynamic programming [13] and randomized proximal methods [14], [15]. An alternative family of model-based tabular PI algorithms with multi-step greedy policy improvement is derived in [16], [17].…”

Optimal Triggering of Networked Control Systems

“…The idea is utilizing the potential of the ADP/RL in handling stochastic processes [45], [53], [54], if the probability distribution functions of the delays and losses are known and using expected value operators in the Bellman equation. While the details are skipped due to the page constraints, interested reader are referred to the available studies both for conventional systems [55], [56] and NCSs [34].…”

“…Therefore, instead of a recursion, one ends up with an equation to solve for the unknown value function. Motivated by the Value Iteration (VI) scheme in ADP/RL for solving conventional problems [45], [56], starting with a guess on V 0 (. ), for example V 0 (.)…”

Proximal algorithms and temporal difference methods for solving fixed point problems

“…which aims to converge to a fixed point of ΠP (c) . The algorithm may be based on simulation-based computations of ΠT (λ) x, and such computations have been discussed in the approximate DP context as part of the LSPE(λ) method (noted earlier), and the λ-policy iteration method (proposed in [BeI96], and further developed in the book [BeT96], and the papers [Ber12b] and [Sch13]). The simulation-based methods for computing ΠT (λ) x have been adapted to the more general linear equation context in [BeY07], [BeY09]; see also [Ber12a], Section 7.3.…”