Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning

Abstract

Learning to reach goal states and learning diverse skills through mutual information (MI) maximization have been proposed as principled frameworks for self-supervised reinforcement learning, allowing agents to acquire broadly applicable multitask policies with minimal reward engineering. Starting from a simple observation that the standard goal-conditioned RL (GCRL) is encapsulated by the optimization objective of variational empowerment, we discuss how GCRL and MIbased RL can be generalized into a single family of methods, which we name variational GCRL (VGCRL), interpreting variational MI maximization, or variational empowerment, as representation learning methods that acquire functionallyaware state representations for goal reaching. This novel perspective allows us to: (1) derive simple but unexplored variants of GCRL to study how adding small representation capacity can already expand its capabilities; (2) investigate how discriminator function capacity and smoothness determine the quality of discovered skills, or latent goals, through modifying latent dimensionality and applying spectral normalization; (3) adapt techniques such as hindsight experience replay (HER) from GCRL to MI-based RL; and lastly, (4) propose a novel evaluation metric, named latent goal reaching (LGR), for comparing empowerment algorithms with different choices of latent dimensionality and discriminator parameterization. Through principled mathematical derivations and careful experimental studies, our work lays a novel foundation from which to evaluate, analyze, and develop representation learning techniques in goal-based RL.

A Controllable Agent by Subgoals in Path Planning Using Goal-Conditioned Reinforcement Learning

“…This paper focuses on how to generate diverse subgoals and how to make the agent learn various subgoals. However, there are a lot of issues to be addressed in the RL as follows [69] :…”

Open-Ended Reinforcement Learning with Neural Reward Functions

“…Our work fits best into the unsupervised skill discovery literature [35,27,20,2,19,46,15]. Compared to DIAYN [19] and similar approaches, we do not set the number of skills to be learnt at the beginning of training.…”

“…Another approach to train multiple behaviours is goal-conditioned learning [29,44,4,43,36,54,58,39,15]. In automated curriculum learning [7,21,23,26,52,22,32,37,40,41,59], a sequence of goals is created such that each of them is not too hard nor too easy for the current agent.…”

Robust Generative Adversarial Imitation Learning via Local Lipschitzness

“…Divergent from these works, we are interested in improving the robustness of the policy learned by GAIL. Recently [22] show that spectral clustering introduced in [19] improves the representation learning capabilities of generative models as it pertains to latent goal discovery in the context of goal-based RL. The Loss-Sensitive Generative Adversarial Network (LS-GAN) [23] induces a Lipschitz regularity condition on the density of real data, i.e., the space of distributions the GAN learns from, which leads to a regularized model which can generate more realistic samples than ordinary GANs.…”

“…Essentially, inequality ( 17) and it's equivalent form (21), are a version of (22) where the transition dynamics are stochastic rather than deterministic. In light of the insights from (22), inequalities (17) and (21) will hold if the stochastic dynamics of the MDP and the corresponding stochastic dynamics of the optimal Markov chain have a property that resembles Lipschitzness of a deterministic function.…”