Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning

Abstract

Efficient exploration remains a challenging problem in reinforcement learning, especially for tasks where extrinsic rewards from environments are sparse or even totally disregarded. Significant advances based on intrinsic motivation show promising results in simple environments but often get stuck in environments with multimodal and stochastic dynamics. In this work, we propose a variational dynamic model based on the conditional variational inference to model the multimodality and stochasticity. We consider the environmental state-action transition as a conditional generative process by generating the nextstate prediction under the condition of the current state, action, and latent variable. We derive an upper bound of the negative log-likelihood of the environmental transition and use such an upper bound as the intrinsic reward for exploration, which allows the agent to learn skills by self-supervised exploration without observing extrinsic rewards. We evaluate the proposed method on several image-based simulation tasks and a real robotic manipulating task. Our method outperforms several state-of-theart environment model-based exploration approaches.

Curiosity-Driven Exploration via Latent Bayesian Surprise

“…Several exploration strategies use a dynamics model to provide intrinsic rewards (Pathak et al 2017;Burda et al 2019b;Houthooft et al 2016;Pathak, Gandhi, and Gupta 2019;Kim et al 2019). Latent variable dynamics have also been studied for exploration (Bai et al 2020;Bucher et al 2019;Tao, Francois-Lavet, and Pineau 2020). Maximum entropy in the state representation has been used for exploration, through random encoders, in RE3 (Seo et al 2021), andprototypical representations, in ProtoRL (Yarats et al 2021).…”

Exploration in Deep Reinforcement Learning: A Comprehensive Survey

“…These works can be technically classified into three categories: 1) methods that estimate prediction errors of the environmental dynamics; 2) methods that estimate the state novelty; 3) methods based on information gain. Table II presents the characteristics of all reviewed intrinsic motivation-oriented ICM [70] high Curiosity-Driven [71] AR4E [106] high VDM [107] high EMI [108] Novelty TRPO-AE-hash [109] partially A3C+ [110] partially DQN-PixelCNN [111] partially φ-EB [112] partially VAE+ME [113] DQN+SR [114] high DORA [115] high A2C+CoEX [65] RND [64] partially Action balance RND [116] partially Informed exploration [117] partially EX 2 [118] high SFC [119] partially partially CB [72] partially high VSIMR [120] high ECO [121] partially high SMM [122] DeepCS [123] Novelty Search [124] high Information gain VIME [125] high AKL [126] high Disagreement [127] high MAX [128] high exploration algorithms in terms of whether it can apply to continuous control problems, and whether it can solve the white-noise and long-horizon problems described in Section III.…”

“…Meanwhile, methods based on prediction errors are limited to deterministic environments, and do not show much advantages in stochastic environments. Later, VDM [107] learns the stochasticity in the transition model through a variational dynamics model, and measures the novelty through evidence lower bound.…”