Natural continual learning: success is a journey, not (just) a destination

Abstract

Biological agents are known to learn many different tasks over the course of their lives, and to be able to revisit previous tasks and behaviors with little to no loss in performance. In contrast, artificial agents are prone to 'catastrophic forgetting' whereby performance on previous tasks deteriorates rapidly as new ones are acquired. This shortcoming has recently been addressed using methods that encourage parameters to stay close to those used for previous tasks. This can be done by (i) using specific parameter regularizers that map out suitable destinations in parameter space, or (ii) guiding the optimization journey by projecting gradients into subspaces that do not interfere with previous tasks. However, parameter regularization has been shown to be relatively ineffective in recurrent neural networks (RNNs), a setting relevant to the study of neural dynamics supporting biological continual learning. Similarly, projection based methods can reach capacity and fail to learn any further as the number of tasks increases. To address these limitations, we propose Natural Continual Learning (NCL), a new method that unifies weight regularization and projected gradient descent. NCL uses Bayesian weight regularization to encourage good performance on all tasks at convergence and combines this with gradient projections designed to prevent catastrophic forgetting during optimization. NCL formalizes gradient projection as a trust region algorithm based on the Fisher information metric, and achieves scalability via a novel Kronecker-factored approximation strategy. Our method outperforms both standard weight regularization techniques and projection based approaches when applied to continual learning problems in RNNs. The trained networks evolve task-specific dynamics that are strongly preserved as new tasks are learned, similar to experimental findings in biological circuits.

The Computational and Neural Bases of Context-Dependent Learning

“…These rotations This information is often explicitly given in the form of task labels (Rusu et al 2016, Aljundi et al 2017, Li & Hoiem 2017, Masse et al 2018, Zeng et al 2019, or sometimes it is more implicit, for example, when only the boundaries between tasks are sign posted, but the identities of individual tasks are not provided (Kirkpatrick et al 2017, Lopez-Paz & Ranzato 2017, Rebuffi et al 2017, Shin et al 2017, Zenke et al 2017. Furthermore, to ensure that each task is learned equally well, several approaches also require networks to be provided with an equal amount of training for each task (Lopez-Paz & Ranzato 2017, Rebuffi et al 2017, Shin et al 2017, Kao et al 2021a). Arguably, this is an unreasonable requirement for a lifelong learning algorithm, as, in general, an equal amount of experience per task cannot be guaranteed in the wild and should not be expected.…”

Logarithmic Continual Learning

“…Particular methods such as SI [10] and EWC [3] vary in the regularization functions. In several works, such as NCL [11], authors combine weights regularization with other techniques, such as gradient projection [12].…”

Biologically-Inspired Continual Learning of Human Motion Sequences

“…Motion is a sequential data type consisting of individual poses, thus a sequential model is required. So far, only a few continual learning studies have tested or developed models for tasks involving sequential or time-series data like text or video, e.g., [13,14,15,16,17,18,19].…”