SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments

Abstract

All living organisms struggle against the forces of nature to carve out a maintainable niche. We propose that such a search for order amidst chaos might offer a unifying principle for the emergence of useful behaviors in artificial agents. We formalize this idea into an unsupervised reinforcement learning method called Surprise Minimizing RL (SMiRL). SMiRL alternates between learning a density model to evaluate the surprise of a stimulus, and improving the policy to seek more predictable stimuli. This process maximizes a lower-bound on the negative entropy of the states, which can be seen as maximizing the agent's ability to maintain order in the environment. The policy seeks out stable and repeatable situations that counteract the environment's prevailing sources of entropy. This might include avoiding other hostile agents, or finding a stable, balanced pose for a bipedal robot in the face of disturbance forces. We demonstrate that our surprise minimizing agents can successfully play Tetris, Doom, control a humanoid to avoid falls, and navigate to escape enemies in a maze without any task-specific reward supervision. We further show that SMiRL can be used together with a standard task rewards to accelerate reward-driven learning.

Adapting Surprise Minimizing Reinforcement Learning Techniques for Transactive Control

“…SMiRL is useful when the environment provides sufficient unexpected and novel events for learning where the challenge for the agent is to maintain a steady equilibrium state. [3] SMiRL maintains a distribution 𝑝 𝜃 (𝑠) about which states are likely under its current policy. The agent then modifies its policy 𝜋 so that it encounters states 𝑠 with high 𝑝 𝜃 (𝑠), as well as to seek out states that will change the model 𝑝 𝜃 (𝑠) so that future states are more likely.…”

“…To account for this we use an augmented MDP that captures this notion. [3] We note that in our implementation of SMiRL, 𝑝 𝜃 𝑡 (𝑠) is normally distributed. To construct the augmented MDP we include sufficient statistics for 𝑝 𝜃 𝑡 (𝑠) in the state space such as the paramaeters of our normal distribution and the number of states seen so far.…”

“…where 𝜇 𝑖 and 𝜎 𝑖 are the sample mean and standard deviation from our state marginal and 𝑠 𝑖 is the 𝑖 𝑡ℎ feature (𝑖 𝑡ℎ hour of day) of 𝑠 [3]. With this formulation we can efficiently calculate the SMiRL reward from our buffer.…”

Skilled motor control implies a low entropy of states but a high entropy of actions

“…Therefore, we allow for lower variability on a perceptual variable (speed indicator) but higher variability on our actions (press or release break) to achieve our control objective. Other related frameworks, such as planning-as-inference, active inference, surprise-minimising RL and KL control are also based on the idea that goal-directed behavior amounts to reducing the entropy or variance of the final (goal) state(s) of the controlled dynamical system [1,4,13,2,31,18]. For example, when balancing an inverted pendulum, the task ends in one single state, which is the one with the pendulum up and still.…”

Show me the Way: Intrinsic Motivation from Demonstrations

“…These many desires shape the way organisms interact with their environment, encouraging them to discover new things but also to protect themselves, avoiding over-surprising events with mechanisms like fear [19]. Berseth et al [5] exemplified how to exploit such priors by implementing a "homeostasis" objective for RL, thereby showing how different from "novelty seeking" these priors can be. Eventually, the resource constraints stop organisms from exploring exhaustively their environment and push them to transfer knowledge from past experience.…”