Implicit Distributional Reinforcement Learning

Abstract

To improve the sample efficiency of policy-gradient based reinforcement learning algorithms, we propose implicit distributional actor critic (IDAC) that consists of a distributional critic, built on two deep generator networks (DGNs), and a semi-implicit actor (SIA), powered by a flexible policy distribution. We adopt a distributional perspective on the discounted cumulative return and model it with a state-action-dependent implicit distribution, which is approximated by the DGNs that take state-action pairs and random noises as their input. Moreover, we use the SIA to provide a semi-implicit policy distribution, which mixes the policy parameters with a reparameterizable distribution that is not constrained by an analytic density function. In this way, the policy's marginal distribution is implicit, providing the potential to model complex properties such as covariance structure and skewness, but its parameter and entropy can still be estimated. We incorporate these features with an off-policy algorithm framework to solve problems with continuous action space, and compare IDAC with the state-of-art algorithms on representative OpenAI Gym environments. We observe that IDAC outperforms these baselines for most tasks.

Exploration by Distributional Reinforcement Learning

“…Our resulting algorithm, E2DC, shows performance gains in difficult continuous control tasks and improvements in distribution matching. Future work includes integrating our changes with orthogonal advances in distributional RL (Kuznetsov et al, 2020) and more expressive policies (Yue et al, 2020;Ward et al, 2019). Due to our method potentially learning more accurate distributions of the true returns, our work here can be leveraged for specific use cases, such as risk-seeking policies in stock market trading strategies, or risk-averse learning for robotics.…”