Autonomous Reinforcement Learning via Subgoal Curricula

Abstract

Reinforcement learning (RL) promises to enable autonomous acquisition of complex behaviors for diverse agents. However, the success of current reinforcement learning algorithms is predicated on an often under-emphasised requirement -each trial needs to start from a fixed initial state distribution. Unfortunately, resetting the environment to its initial state after each trial requires substantial amount of human supervision and extensive instrumentation of the environment which defeats the purpose of autonomous reinforcement learning. In this work, we propose Value-accelerated Persistent Reinforcement Learning (VaPRL), which generates a curriculum of initial states such that the agent can bootstrap on the success of easier tasks to efficiently learn harder tasks. The agent also learns to reach the initial states proposed by the curriculum, minimizing the reliance on human interventions into the learning. We observe that VaPRL reduces the interventions required by three orders of magnitude compared to episodic RL while outperforming prior state-of-the art methods for reset-free RL both in terms of sample efficiency and asymptotic performance on a variety of simulated robotics problems 1 .

Autonomous Reinforcement Learning: Formalism and Benchmarking

“…We evaluate forward-backward RL (FBRL) (Han et al, 2015;Eysenbach et al, 2017), a perturbation controller (R3L) (Zhu et al, 2020), value-accelerated persistent RL (VaPRL) (Sharma et al, 2021), a comparison to simply running the base RL algorithm with the biased TD update discussed in Section 6.1 (naïve RL), and finally an oracle (oracle RL) where resets are provided are provided every H E steps (H T is typically three orders of magnitude larger than H E ). We benchmark VaPRL only when demonstrations are available, in accordance to the proposed algorithm in Sharma et al (2021). We average the performance of all algorithms across 5 random seeds.…”

“…Reset-free RL has been studied by previous works with a focus on safety (Eysenbach et al, 2017), automated and unattended learning in the real world (Han et al, 2015;Zhu et al, 2020;, skill discovery Lu et al, 2020), and providing a curriculum (Sharma et al, 2021). Strategies to learn reset-free behavior include directly learning a backward reset controller (Eysenbach et al, 2017), learning a set of auxillary tasks that can serve as an approximate reset (Ha et al, 2020;, or using a novelty seeking reset controller (Zhu et al, 2020).…”

“…A.1 GOAL-CONDITIONED ARL This framework can be readily extended to goal-conditioned reinforcement learning (Kaelbling, 1993;Schaul et al, 2015), which is also an area of study covered in some prior works on resetfree reinforcement learning (Sharma et al, 2021). Assuming a goal space G and task-distribution p g : G → R ≥0 , assume that the algorithm A : {s i , a i , s i+1 } t−1 i=0 → (a t , π t ), where a t ∈ A and π t : S × A × G → R ≥0 .…”

“…The first is a setting where the agent first trains in a non-episodic environment, and is then "deployed" into an episodic test environment. In this setting, which is most commonly studied in prior works on "resetfree" learning (Han et al, 2015;Zhu et al, 2020;Sharma et al, 2021), the goal is to learn the best possible episodic policy after a period of non-episodic training. For instance, in the case of a home cleaning robot, this would correspond to evaluating its ability to clean a messy home.…”

“…The main contributions of our work consist of a benchmark for autonomous RL (ARL), as well as formal definitions of two distinct ARL settings. Our benchmarks combine components from previously proposed environments (Coumans & Bai, 2016;Gupta et al, 2019;Yu et al, 2020;Sharma et al, 2021), but reformulate the learning tasks to reflect ARL constraints, such as the absence of explicitly available resets. Our formalization of ARL relates it to the standard RL problem statement, provides a concrete and general definition, and provides a number of instantiations that describe how common ingredients of ARL, such as irreversible states, interventions, and other components, fit into the general framework.…”