URLB: Unsupervised Reinforcement Learning Benchmark

Abstract

Deep Reinforcement Learning (RL) has emerged as a powerful paradigm to solve a range of complex yet specific control tasks. Yet training generalist agents that can quickly adapt to new tasks remains an outstanding challenge. Recent advances in unsupervised RL have shown that pre-training RL agents with self-supervised intrinsic rewards can result in efficient adaptation. However, these algorithms have been hard to compare and develop due to the lack of a unified benchmark. To this end, we introduce the Unsupervised Reinforcement Learning Benchmark (URLB). URLB consists of two phases: reward-free pre-training and downstream task adaptation with extrinsic rewards. Building on the DeepMind Control Suite, we provide twelve continuous control tasks from three domains for evaluation and open-source code for eight leading unsupervised RL methods. We find that the implemented baselines make progress but are not able to solve URLB and propose directions for future research. Code for the benchmark and implemented baselines can be accessed at https://github.com/rll-research/url_benchmark. * equal contribution, order determined by coin flip.35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks.

Physical Derivatives: Computing policy gradients by physical forward-propagation

“…However, reward binds the agent to a certain task for which the reward represents success. Aligned with the recent surge of interest in unsupervised methods in reinforcement learning (Baranes and Oudeyer, 2013;Bellemare et al, 2016;Gregor et al, 2016;Houthooft et al, 2016;Gupta et al, 2018;Hausman et al, 2018;Pong et al, 2019;Laskin et al, 2020Laskin et al, , 2021He et al, 2021) and previously proposed ideas (Schmidhuber, 1991a(Schmidhuber, , 2010, we argue that there exist properties of a dynamical system which are not tied to any particular task, yet highly useful, leveraging them can help solve other tasks more efficiently. This work focuses on the sensitivity of the produced trajectories of the system with respect to the policy so-called Physical Derivatives.…”

CUDC: A Curiosity-Driven Unsupervised Data Collection Method with Adaptive Temporal Distances for Offline Reinforcement Learning

“…Competence-based models like DIAYN (Eysenbach et al 2019), SMM (Lee et al 2019), and APS (Liu and Abbeel 2021a) encourage agents to learn diverse skills by leveraging prior information. However, all of these methods were originally designed for online pretraining and fine-tuning (Laskin et al 2021), not tailored for data collection. In contrast, CUDC is a novel method that gradually expands the feature space by exploiting reachability into more distant future states, rather than a fixed temporal distance.…”

Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping

“…In particular, Yarats et al [7] creates a dataset of pre-collected trajectories, ExoRL, on the DeepMind control suite [29] generated without any hand-crafted rewards. Similar to URLB [30], ExoRL benchmarks a number of exploration algorithms [3,6,31,5], and evaluates the performance of a policy trained on the corresponding offline datasets relabeled with task-specific rewards.…”