Fast and Efficient Reinforcement Learning with Truncated Temporal Differences

Cichosz, Paweł; Mulawka, Jan J.

doi:10.1016/b978-1-55860-377-6.50021-9

Cited by 15 publications

(16 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the number of state-action pairs for which actual updating is required can be kept at a manageable level by maintaining only those state-action pairs whose activity trace (3,A) '~ is significant, since this quantity declines exponentially when "yA < 1. For a more elaborate procedure see Cichosz & Mulawka (1995). Another approach is to implement a Q(A)-learning system on a parallel machine in which each state-action pair is mapped onto a separate processor.…”

Section: R~ -(~T(xt At) = E' T + 7)~et+l + 72/~2et+2 +"" (X)mentioning

confidence: 99%

Incremental multi-step Q-learning

1996

View full text Add to dashboard Cite

Abstract. This paper presents a novel incremental algorithm that combines Q-learning, a well-known dynamicprogramming based reinforcement learning method, with the TD(A) return estimation process, which is typically used in actor-critic learning, another well-known dynamic-programming based reinforcement learning method. The parameter A is used to distribute credit throughout sequences of actions, leading to faster learning and also helping to alleviate the non-Markovian effect of coarse state-space quantization. The resulting algorithm, Q(A)-learning, thus combines some of the best features of the Q-learning and actor-critic learning paradigms. The behavior of this algorithm has been demonstrated through computer simulations.

show abstract

Section: R~ -(~T(xt At) = E' T + 7)~et+l + 72/~2et+2 +"" (X)mentioning

confidence: 99%

Incremental multi-step Q-learning

1996

View full text Add to dashboard Cite

show abstract

“…For all algorithms 7 was set to 0.95 and T to 0.01. These values appeared reasonable based on prior work with TTD [2,4].…”

Section: Experimental Design and Resultsmentioning

confidence: 90%

“…This can be performed either iteratively, based directly on the definition of TTD returns, or in an incremental manner, which is particularly efficient. The appropriate algorithms are described in detail in the existing TTD literature [2,4].…”

Section: Ttd Proceduresmentioning

confidence: 99%

“…The TTD procedure [2,4] allows one to implement TD-based reinforcement learning algorithms in their TD()~ > 0) versions without conceptually appealing, but computationally demanding eligibility traces [1,10,4]. Only its particular instantiation for the Q-learning algorithm is discussed below, but modifications for other RL algorithms are straightforward [2].…”

Section: Ttd Proceduresmentioning

confidence: 99%

“…Figure 5 into, respectively, 22 = 4 and 52 = 25 equal smaller cells. The agent's starting location is at (2, 6) for the 10 x 10 task, at (4,12) for the 20 x 20 task, and at (12, 32) for the 50 x 50 task, assuming that the coordinates count from 0 and start from the left upper corner. The shortest path to any goal cell for the three tasks consists, respectively, of 20 steps, 36 steps, and 84 steps.…”

Section: Learning Tasksmentioning

confidence: 99%

See 2 more Smart Citations

Integrated learning and planning based on truncating temporal differences

Cichosz

1997

Machine Learning: ECML-97

View full text Add to dashboard Cite

Abstract. Reinforcement learning systems learn to act in an uncertain environment by executing actions and observing their long-term effects. A large number of time steps may be required before this trial-and-error process converges to a satisfactory policy. It is highly desirable that the number of experiences needed by the system to learn to perform its task be minimized, particularly if making errors costs much. One approach to achieve this goal is to use hypothetical experiences, which requires some additional computation, but may reduce the necessary number of much more costly real experiences. This well-known idea of augmenting reinforcement learning by planning is revisited in this paper in the context of truncated TD(A), or TTD, a simple computational technique which allows reinforcement learning algorithms based on the methods of temporal differences to learn considerably faster with essentially no additional computational expense. Two different ways of combining TTD with planning are proposed which make it possible to benefit from A > 0 in both the learning and planning processes. The algorithms are evaluated experimentally on a family of grid path-finding tasks and shown to indeed yield a considerable reduction of the number of real interactions with the environment necessary to converge, as well as an improvement of scaling properties.

show abstract

Technical Note

Peng

Williams

Recent Advances in Reinforcement Learning

View full text Add to dashboard Cite

Fast and Efficient Reinforcement Learning with Truncated Temporal Differences

Cited by 15 publications

References 10 publications

Incremental multi-step Q-learning

Incremental multi-step Q-learning

Integrated learning and planning based on truncating temporal differences

Technical Note

Contact Info

Product

Resources

About