DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning

Abstract

This paper prescribes a suite of techniques for off-policy Reinforcement Learning (RL) that simplify the training process and reduce the sample complexity. First, we show that simple Deterministic Policy Gradient works remarkably well as long as the overestimation bias is controlled. This is contrast to existing literature which creates sophisticated off-policy techniques. Second, we pinpoint training instabilities, typical of off-policy algorithms, to the greedy policy update step; existing solutions such as delayed policy updates do not mitigate this issue. Third, we show that ideas in the propensity estimation literature can be used to importance-sample transitions from the replay buffer and selectively update the policy to prevent deterioration of performance. We make these claims using extensive experimentation on a set of challenging MuJoCo tasks. A short video of our results can be seen at https://tinyurl.com/scs6p5m.

Continuous Doubly Constrained Batch Reinforcement Learning

“…There are several strategies to address this issue (Lan et al, 2020;Fujimoto et al, 2018;Fakoor et al, 2020;Hasselt et al, 2016;Hasselt, 2010), albeit with varying degrees of success. propose a straightforward approach via a convex combination of the extremes of the Q distribution.…”