Overcoming catastrophic forgetting in neural networks

Abstract

The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.synaptic consolidation | artificial intelligence | stability plasticity | continual learning | deep learning A chieving artificial general intelligence requires that agents are able to learn and remember many different tasks (1). This is particularly difficult in real-world settings: The sequence of tasks may not be explicitly labeled, tasks may switch unpredictably, and any individual task may not recur for long time intervals. Critically, therefore, intelligent agents must demonstrate a capacity for continual learning: that is, the ability to learn consecutive tasks without forgetting how to perform previously trained tasks.Continual learning poses particular challenges for artificial neural networks due to the tendency for knowledge of the previously learned task(s) (e.g., task A) to be abruptly lost as information relevant to the current task (e.g., task B) is incorporated. This phenomenon, termed catastrophic forgetting (2-6), occurs specifically when the network is trained sequentially on multiple tasks because the weights in the network that are important for task A are changed to meet the objectives of task B. Whereas recent advances in machine learning and in particular deep neural networks have resulted in impressive gains in performance across a variety of domains (e.g., refs. 7 and 8), little progress has been made in achieving continual learning. Current approaches have typically ensured that data from all tasks are simultaneously available during training. By interleaving data from multiple tasks during learning, forgetting does not occur because the weights of the network can be jointly optimized for performance on all tasks. In this regime-often referred to as the multitask learning paradigm-deep-learning techniques have been used to train single agents that can successfully play multiple Atari games (9, 10). If tasks are presented sequentially, multitask learning can be used only if the data are recorded by an episodic memory system and replayed to the network during training. This approach [often called system-level consolidation (4, 5)] is impractical for learning large numbers of tasks, as in our setting it would require the amount of memories being stored and replayed to be proportional to the number of tasks. The lack of algorithms to support continual learning thus rema...

Visual pathways from the perspective of cost functions and multi-task deep neural networks

“…Similarly, various properties can be derived from a single stream, like visual aspects (depth, figure-ground separation, segmentation), from an object recognition stream, where each aspect sub-stream is learned via a separate cost function. For tasks sharing outputs, and thus having overlap over different tasks, evidence increasingly suggests that the brain selectively "protects" synapses for modification by new tasks, effectively "unsharing" these parameters between tasks (Kirkpatrick et al, 2017).…”

The End of Moore’s Law: Opportunities for Natural Computing?

“…Such a weak causality indicates novelty of information, and thus encourages the brain to employ knowledge about past experiences in its learning of the new information. Artificial neural networks, on the other hand, face difficulties in this task, because new information will interfere with previously learned knowledge [30]. Though deep learning excels in finding intricate structures in big data sets, it does less well than the brain when only small quantities of training data are available.…”

Guided Policy Search for Sequential Multitask Learning

“…However, current neural network approaches have still not been able to fully implement continual learning, and there is also inevitable catastrophic forgetting associated with this mode of learning. In an attempt to enable agents to continuously learn without catastrophic forgetting, James et al [26] recently proposed training of networks, using an elastic weight consolidation (EWC) algorithm, that can maintain expertise from previously learnt tasks. For gradient policy learning, a deep deterministic-policy gradient (DDPG) algorithm [27] has been proposed to continuously improve policy, whilst an agent explores its environment.…”

“…As noted earlier, current GPS approaches can only handle scenarios where data from all tasks is simultaneously made available during the early training stage, which constitutes an impractical constraint for consecutive task learning. By exploiting and adapting the recently developed EWC algorithm [26], we propose incorporation of Fisher information, to protect weights that are important for previous tasks, while learning the new task at hand. To some extent, this also overcomes catastrophic forgetting, in our proposed approach to sequential multi-task learning.…”