Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations

Abstract

Currently, deep reinforcement learning (RL) shows impressive results in complex gaming and robotic environments. Often these results are achieved at the expense of huge computational costs and require an incredible number of episodes of interaction between the agent and the environment. There are two main approaches to improving the sample efficiency of reinforcement learning methods -using hierarchical methods and expert demonstrations. In this paper, we propose a combination of these approaches that allow the agent to use low-quality demonstrations in complex vision-based environments with multiple related goals. Our forgetful experience replay (ForgER) algorithm effectively handles errors in expert data and reduces quality losses when adapting the action space and states representation to the agent's capabilities. Our proposed goal-oriented structuring of replay buffer allows the agent to automatically highlight sub-goals for solving complex hierarchical tasks in demonstrations. Our method is universal and can be integrated into various off-policy methods. It surpasses all known existing state-of-the-art RL methods using expert demonstrations on various model environments. The solution based on our algorithm beats all the solutions for the famous MineRL competition and allows the agent to mine a diamond in the Minecraft environment.Preprint. Under review.

Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos

“…Minecraft is a voxel-based 3D video game that, due its popularity and wide variety of mechanics, has attracted a vast amount of RL research. 27,28,[30][31][32][33][34][52][53][54][55][56][57][58][59][60] A large body of work focuses on small, custom-made Minecraft worlds with tasks such as navigation, 53,60 block placing, 54,55 instruction following, 58,59 combat, 56 and others. 28,31,57 Work operating in the massive, randomly generated environments of Minecraft itself has included hill climbing, 52 automated curriculum learning 30 and, most closely related to the RL experiments presented in Sec.…”

“…4.4, diamond mining. 27,[32][33][34] However, to the best of our knowledge, there is no published work that operates in the full, unmodified human action space, which includes drag-and-drop inventory management and item crafting.…”

“…Others have managed to obtain diamonds with ∼ 0.1% reliability in 15 minutes 32,33 but always with a simplified action space designed to ease exploration. To the best of our knowledge, we are the first to report non-zero success rates on crafting a diamond pickaxe.…”

“…1). Compared to prior work in Minecraft that uses a lower frame rate and constructs crafting and attacking macros, 30,[32][33][34] using the native human interface drastically increases the environment's exploration difficulty, making most simple tasks near impossible with RL from scratch. Even the simple task of gathering a single wooden log while already facing a tree takes 60 consecutive attack actions with the human interface, meaning the chance for a naive random policy to succeed is 1 2 60 .…”

Theoretical Remarks on Feudal Hierarchies and Reinforcement Learning

“…While the most well-known frameworks that give rise and exploit hierarchies on sequential decision making problems remain virtually unchanged [3,36,4,28], their use is both expanding and diversifying [40,15,2]. As an example that attests for its power, hierarchies and reinforcement learning, combined with expert demonstrations, set the state of the art on the game of Minecraft [7,20,33,24]. Hierarchical reinforcement learning is also piercing through the multi-agent field [23,14,1].…”