TD-regularized actor-critic methods

Abstract

Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and the critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize the learning. To avoid such issues, we propose to regularize the learning objective of the actor by penalizing the temporal difference (TD) error of the critic. This improves stability by avoiding large steps in the actor update whenever the critic is highly inaccurate. The resulting method, which we call the TD-regularized actor-critic method, is a simple plug-and-play approach to improve stability and overall performance of the actor-critic methods. Evaluations on standard benchmarks confirm this. Source code can be found at https://github.com/sparisi/td-reg 1 Convergence is assured when special types of linear functions known as compatible functions are used to model the critic [Sutton et al., 1999, Peters andSchaal, 2008]. Convergence for other types of approximators is assured only for some algorithms and under some assumptions [Baird, 1995, Konda and Tsitsiklis, 2000, Castro et al., 2008.

Safe Driving Of Autonomous Vehicles Through Improved Deep Reinforcement Learning

“…Action-state based inference along with the ability to optimize value-functions and policies in separate, iterative steps has led to variational expectation-maximization in actor-critic algorithms [56]. Our experiments demonstrate that actor-critic algorithms in conjunction with VAE outperform the state-of-the-art methods based solely on soft-actor critic or deterministic value-functions in self-driving domain.…”

“…If a random variable can be any real number with equal probability then it is highly unpredictable and has very high entropy [60]. A high entropy in policy encourages exploration, and assigns equal probabilities to actions that have same or nearly equal Q-values [56]. It ensures that exploration does not collapse into repeatedly selecting a particular action leading to inconsistency in the approximated Q-function by assigning a high probability to any one action out of the possible set of actions [42].…”

Policy-Gradient and Actor-Critic Based State Representation Learning for Safe Driving of Autonomous Vehicles

“…In the existing literature, the driving environment is usually Rayleigh distributed [ 8 ]. Actor-critic methods have achieved incredible performance on RL problems such as games, but they are prone to instability due to frequent interaction between the actor and critic during learning [ 7 ]. An inaccurate step taken at one stage might adversely affect the subsequent steps, destabilizing the learning.…”

“…Deep reinforcement learning has been widely applied to various problems, predominantly in game playing [ 7 , 8 ]. Deep reinforcement learning has also been extensively applied to resource allocation and channel estimation problems in wireless communication, autonomous routing and self-healing in networking, localization and path-planning in unmanned air vehicles (UAV), smart-drones and underwater communications.…”

“…The reward r, given by (7) prevents the vehicle from deviating off the track while allowing the vehicle to maintain its position on the road [24].…”