Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration

Abstract

Exploration in sparse reward reinforcement learning remains an open challenge. Many state-of-the-art methods use intrinsic motivation to complement the sparse extrinsic reward signal, giving the agent more opportunities to receive feedback during exploration. Commonly these signals are added as bonus rewards, which results in a mixture policy that neither conducts exploration nor task fulfillment resolutely. In this paper, we instead learn separate intrinsic and extrinsic task policies and schedule between these different drives to accelerate exploration and stabilize learning. Moreover, we introduce a new type of intrinsic reward denoted as successor feature control (SFC), which is general and not task-specific. It takes into account statistics over complete trajectories and thus differs from previous methods that only use local information to evaluate intrinsic motivation. We evaluate our proposed scheduled intrinsic drive (SID) agent using three different environments with pure visual inputs: VizDoom, DeepMind Lab and DeepMind Control Suite. The results show a substantially improved exploration efficiency with SFC and the hierarchical usage of the intrinsic drives. A video of our experimental results can be found at https://youtu.be/b0MbY3lUlEI.

BeBold: Exploration Beyond the Boundary of Explored Regions

“…Asymptotic Inconsistency. Approaches that define IR as the difference between state representations ψ(s) − ψ(s ) (ψ is a learned embedding network) (Zhang et al, 2019;Marino et al, 2019) suffer from asymptotic inconsistency. In other words, their IR does not vanish even after sufficient exploration: r i → 0 when N → ∞.…”

“…For this, (Zhang et al, 2019) proposes to learn a separate scheduler to switch between intrinsic and extrinsic rewards, and divides the state representation difference by the square root of visitation counts. In comparison, BeBold does not require any extra stage and is a much simpler solution.…”

“…Recent approaches have proposed to use intrinsic rewards (IR) (Schmidhuber, 2010) to motivate agents for exploration before any extrinsic rewards are obtained. Various criteria have been proposed, including curiosity/surprise-driven (Pathak et al, 2017), count-based (Bellemare et al, 2016;Burda et al, 2018a;b;Badia et al, 2020b), and state-diff approaches (Zhang et al, 2019;Marino et al, 2019).…”

Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning

“…After training the pure-exploratory policy with intrinsic rewards, there are several ways to combine the intrinsic policy with extrinsic policies that are trained by other extrinsic rewards to enhance the performance. Scheduled intrinsic drive [34] uses a high-level scheduler that periodically selects to follow either the extrinsic or the intrinsic policy to gather experiences. MuleX [35] learns several policies independently and uses a random heuristic to decide which one to use in each time step.…”

Scaling MAP-Elites to Deep Neuroevolution

“…While this might work when the two objectives are somewhat aligned, it may be inefficient when they are not [41]. Several works have started to investigate this question and some propose to disentangle exploration and exploitation into distinct phases [7,11,53]. QD presents a natural way of decoupling the optimization of exploitation (quality) and exploration (diversity) by looking for high-performing solutions in local niches of the behavioral space, leading to local competition between solutions instead of a global competition [14,32,35].…”