Exploration by Random Network Distillation

Abstract

We introduce an exploration bonus for deep reinforcement learning methods that is easy to implement and adds minimal overhead to the computation performed. The bonus is the error of a neural network predicting features of the observations given by a fixed randomly initialized neural network. We also introduce a method to flexibly combine intrinsic and extrinsic rewards. We find that the random network distillation (RND) bonus combined with this increased flexibility enables significant progress on several hard exploration Atari games. In particular we establish state of the art performance on Montezuma's Revenge, a game famously difficult for deep reinforcement learning methods. To the best of our knowledge, this is the first method that achieves better than average human performance on this game without using demonstrations or having access to the underlying state of the game, and occasionally completes the first level.

Deep Graph-level Anomaly Detection by Glocal Knowledge Distillation

“…However, for learning tasks on graph-level data, no such general-purpose pretrained teacher networks are available; further, graph databases from different domains differ significantly from each other, which also prevents the application of this type of approach to the GAD task. Random knowledge distillation is originally introduced in [5] to address sparse reward problems in deep reinforcement learning (DRL). It uses the random distillation errors to measure the novelty of states as some additional reward signals to encourage DRL agents' exploration in sparse-reward contexts.…”

“…The aim is to calculate the posterior after iteratively updating on the data. According to [5], our task can then be formulated as the optimization problem below:…”

“…is equivalent to distilling a randomly drawn function from the prior. From this perspective, each entry of the representation outputs of the target and the predictor networks would correspond to a part of an ensemble and the prediction error would be an estimate of the predictive variance of the ensemble when the ensemble is assumed to be unbiased, as discussed in [5]. If we consider 𝜙 (•, Θ ★ ) as the target network with randomly initialized Θ ★ and regard 𝜙 (•, Θ) as the predictor network, the prediction errors of the node…”

“…Adam is the default optimizer used in the above three methods. Both WL and PK are directly taken from the GraKeL library 0.1.8 5 . For WL, we perform three iterations to obtain the graph representations, which utilize the same neighborhood information as a three-layer GCN as in our model and OCGCN.…”

JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning

“…Existing methods use curiosity or uncertainty as a signal for exploration [Pathak et al, 2017;Burda et al, 2018] so that the learned agent is able to cover a large state space. However, the exploration-exploitation dilemma, given the sample efficiency consideration, drives us to develop self-imitation learning (SIL) [Oh et al, 2018] methods that focus on exploiting past good experiences for better exploration.…”