S-TRIGGER: Continual State Representation Learning via Self-Triggered Generative Replay

Abstract

We consider the problem of building a state representation model for control, in a continual learning setting. As the environment changes, the aim is to efficiently compress the sensory state's information without losing past knowledge, and then use Reinforcement Learning on the resulting features for efficient policy learning. To this end, we propose S-TRIGGER, a general method for Continual State Representation Learning applicable to Variational Auto-Encoders and its many variants. The method is based on Generative Replay, i.e. the use of generated samples to maintain past knowledge. It comes along with a statistically sound method for environment change detection, which self-triggers the Generative Replay. Our experiments on VAEs show that S-TRIGGER learns state representations that allows fast and high-performing Reinforcement Learning, while avoiding catastrophic forgetting. The resulting system is capable of autonomously learning new information without using past data and with a bounded system size. Code for our experiments is attached in Appendix.

Replay in Deep Learning: Current Approaches and Missing Biological Elements

“…Specifically, the Exemplar-Supported Generative Reproduction model (He et al, 2018) uses a GAN to generate pseudoexamples for replay during continual learning, while the Dynamic Generative Mem-ory model (Ostapenko et al, 2019), the Deep Generative Replay model (Shin et al, 2017), the Memory Replay GAN model (Wu et al, 2018), and the Closed-Loop GAN model (Rios and Itti, 2018) are all used to continually learn to generate images and scenes. Continual learning with replay in GANs has also been used for reinforcement learning (Caselles-Dupré et al, 2019). Moreover, unsupervised learning techniques such as auto-encoders and GANs are widely used to generate replay samples in supervised learning algorithms (Draelos et al, 2017;.…”

Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting

“…Furthermore, the input items RePR generates are not static images but rather a sequence of consecutive frames. Since our work, pseudo-rehearsal has been used to overcome CF in models which have learnt to generate states from previously seen environments [39], [40]. In both these cases, pseudo-rehearsal was not applied to the learning agent to prevent its CF.…”

Symmetry-Based Disentangled Representation Learning requires Interaction with Environments

“…training procedure is presented in Algorithm 1 in Appendix C.2 For each image in a batch, we compute f (o t ) = z t and f (o t+1 ) = z t+1 using the encoder part of the VAE. Then we decode z t with the decoder and compute the reconstruction loss L reconstruction and annealed KL divergence L KL as in(Caselles-Dupré et al, 2019). Then we compute Â(a t ) • z t and compute the forward loss, which is the MSE with z t+1 : L f orward = ( Â(a t ) • z t − z t+1 ) 2 .…”