Distributed Prioritized Experience Replay

Abstract

We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

Deep Reinforcement Learning for Radio Resource Allocation and Management in Next Generation Heterogeneous Wireless Networks: A Survey

“…Other variants of the value-based algorithms are developed to enhance the performance of vanilla DQN algorithm in terms of stability, convergence speed, implementation complexity, sample/learning efficiency, etc. Such variants include prioritized experience replay DQN [75], distributed prioritized experience replay DQN [76], distributional DQN [77], Rainbow DQN [78], and recurrent DQN [79].…”

CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms

“…For this motivation, we have also implemented dqn.py, dqn atari.py (Mnih et al, 2013), c51.py, c51 atari.py (Bellemare et al, 2017), apex atari.py (Horgan et al, 2018), ddpg continuous action.py (Lillicrap et al, 2015), td3 continuous action.py (Fujimoto et al, 2018), and sac continuous action.py (Haarnoja et al, 2018).…”

Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration

“…There are some similar properties between prediction error and TD error: 1) they both converge when the policy converges; 2) they are common metrics that show promising results for exploration [12,11,42,16] and exploitation [43][44][45], respectively. These similarities motivate us to study the effects of EMC's each module by comparing with the ablation study using TD error.…”