Curiosity-Driven Exploration via Latent Bayesian Surprise

Abstract

The human intrinsic desire to pursue knowledge, also known as curiosity, is considered essential in the process of skill acquisition. With the aid of artificial curiosity, we could equip current techniques for control, such as Reinforcement Learning, with more natural exploration capabilities. A promising approach in this respect has consisted of using Bayesian surprise on model parameters, i.e. a metric for the difference between prior and posterior beliefs, to favour exploration. In this contribution, we propose to apply Bayesian surprise in a latent space representing the agent’s current understanding of the dynamics of the system, drastically reducing the computational costs. We extensively evaluate our method by measuring the agent's performance in terms of environment exploration, for continuous tasks, and looking at the game scores achieved, for video games. Our model is computationally cheap and compares positively with current state-of-the-art methods on several problems. We also investigate the effects caused by stochasticity in the environment, which is often a failure case for curiosity-driven agents. In this regime, the results suggest that our approach is resilient to stochastic transitions.

Self-mediated exploration in artificial intelligence inspired by cognitive psychology

“…Studies exploring emotion in AI seldom consider both a robust psychological and a neurophysiological bases for its formulation, with fewer even employing it as a driver of learning factors. Regardless, our work relates with literature such as [21], wherein Mazzaglia et al developed a latent dynamics model endowed with Bayesian surprise as the dissimilarity from its posterior to prior beliefs, which rewards exploration when occurring. Schillaci et al also presented a process for estimating the change in prediction error (PE) as a metric of learning progress [26].…”

Latent Learning Progress Drives Autonomous Goal Selection in Human Reinforcement Learning

“…KB intrinsic motivations derive from comparisons between pre-existing and newly acquired information and are well-documented in the animal world (e.g., [1,33,34,3,35,6,5,36,37,8]). In artificial agents, KB signals of novelty, diversity, or prediction error broaden exploration when provided alongside extrinsic rewards [13,15,38,16,39] or even on their own ( [14,11], see [40] for review). However, human curiosity is often driven towards stimuli of intermediate complexity [41,42,7], rather than extremes.…”

The Free Energy Principle for Perception and Action: A Deep Learning Perspective

“…State-driven Exploration. Maximizing mutual information between states and observations has also been studied in RL for exploration, using the Bayesian surprise signal given by the D KL divergence between the (autoencoding) posterior and the prior of the model as a reward [51]. Alternatively, the surprisal with respect to future observations has also been used in RL to generate an intrinsic motivation signal that rewards exploration [148,52,149].…”

“…In deep learning, generative models have been widely studied, obtaining outstanding results in several domains, such as image generation [38,39,40], text prediction [41,42,43], and video modeling [44,45,46,47]. In particular, temporal deep generative models that allow predicting the dynamics of a system, i.e., the environment or world, have been studied for control [48,49,50], curiosity and exploration [51,52,53], and anomaly detection [54]. Several of these models have been used in settings that are similar to the active inference one, and some of them even share some similarities with the active inference objective of minimizing variational free energy.…”