Truncating Temporal Differences: On the Efficient Implementation of TD(lambda) for Reinforcement Learning

Abstract

Temporal di erence (TD) methods constitute a class of methods for learning predictions in multi-step prediction problems, parameterized by a recency factor . Currently the most important application of these methods is to temporal credit assignment in reinforcement learning. Well known reinforcement learning algorithms, such as AHC or Q-learning, may be viewed as instances of TD learning. This paper examines the issues of the e cient and general implementation of TD( ) for arbitrary , for use with reinforcement learning algorithms optimizing the discounted sum of rewards. The traditional approach, based on eligibility traces, is argued to su er from both ine ciency and lack of generality. The TTD (Truncated Temporal Di erences) procedure is proposed as an alternative, that indeed only approximates TD( ), but requires very little computation per action and can be used with arbitrary function representation methods. The idea from which it is derived is fairly simple and not new, but probably unexplored so far. Encouraging experimental results are presented, suggesting that using > 0 with the TTD procedure allows one to obtain a signi cant learning speedup at essentially the same cost as usual TD(0) learning.

Integrated learning and planning based on truncating temporal differences

“…The TTD procedure [2,4] allows one to implement TD-based reinforcement learning algorithms in their TD()~ > 0) versions without conceptually appealing, but computationally demanding eligibility traces [1,10,4]. Only its particular instantiation for the Q-learning algorithm is discussed below, but modifications for other RL algorithms are straightforward [2].…”

“…Only its particular instantiation for the Q-learning algorithm is discussed below, but modifications for other RL algorithms are straightforward [2].…”

“…TTD relies on applying, at time t, the following update operation to the state and action from time t -m + h A~rn update ~ (Q, Xt--m+l; at-m+1, Zt_m+ I --Q(xt-m+l, at-m+1)), (2) where zt ~'m is the TTD()~, m) return for time t, defined as . This can be performed either iteratively, based directly on the definition of TTD returns, or in an incremental manner, which is particularly efficient.…”

“…This can be performed either iteratively, based directly on the definition of TTD returns, or in an incremental manner, which is particularly efficient. The appropriate algorithms are described in detail in the existing TTD literature [2,4].…”

“…The last approach is investigated in this paper in a new, promising context. Novel hybrid learning and planning algorithms are proposed, obtained by combining the frameworks of the Dyna architecture [12] and of the TTD procedure [2]. These two are briefly described in the next section.…”