Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning

Caccia, M.; Rodríguez, Paul; Ostapenko, Оleksiy; Fabrice, Normandin,; Lin, Min; Caccia, Lucas; Laradji, Issam; Rish, Irina; Lacoste, Alexandre; Vázquez, David; Charlin, Laurent

doi:10.48550/arxiv.2003.05856

Cited by 10 publications

(13 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The agents can even extrapolate far beyond the training distribution to recall words after subsequent learning episodes without ever being trained to do so. Thus better memory systems can help with the challenge of bridging from meta-learning to continual learning, which is receiving increasing interest [12,16,43,4,36].…”

Section: Discussionmentioning

confidence: 99%

Towards mental time travel: a hierarchical memory for reinforcement learning agents

Lampinen¹,

Chan²,

Banino³

et al. 2021

Preprint

View full text Add to dashboard Cite

Reinforcement learning agents often forget details of the past, especially after delays or distractor tasks. Agents with common memory architectures struggle to recall and integrate across multiple timesteps of a past event, or even to recall the details of a single timestep that is followed by distractor tasks. To address these limitations, we propose a Hierarchical Transformer Memory (HTM), which helps agents to remember the past in detail. HTM stores memories by dividing the past into chunks, and recalls by first performing high-level attention over coarse summaries of the chunks, and then performing detailed attention within only the most relevant chunks. An agent with HTM can therefore "mentally time-travel"-remember past events in detail without attending to all intervening events. We show that agents with HTM substantially outperform agents with other memory architectures at tasks requiring long-term recall, retention, or reasoning over memory. These include recalling where an object is hidden in a 3D environment, rapidly learning to navigate efficiently in a new neighborhood, and rapidly learning and retaining new object names. Agents with HTM can extrapolate to task sequences an order of magnitude longer than they were trained on, and can even generalize zero-shot from a meta-learning setting to maintaining knowledge across episodes. HTM improves agent sample efficiency, generalization, and generality (by solving tasks that previously required specialized architectures). Our work is a step towards agents that can learn, interact, and adapt in complex and temporally-extended environments.Preprint. Under review.

show abstract

Section: Discussionmentioning

confidence: 99%

Towards mental time travel: a hierarchical memory for reinforcement learning agents

Lampinen¹,

Chan²,

Banino³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In contrast, our method does not even attempt to find the task boundaries, but directly adapts without them. A number of related works also address continual learning via meta-learning, but with the aim of minimizing catastrophic forgetting Gupta et al (2020); Caccia et al (2020). Our aim is not to address catastrophic forgetting.…”

Section: Related Workmentioning

confidence: 99%

Fully Online Meta-Learning Without Task Boundaries

Rajasegaran¹,

Finn²,

Levine³

2022

Preprint

View full text Add to dashboard Cite

While deep networks can learn complex functions such as classifiers, detectors, and trackers, many applications require models that continually adapt to changing input distributions, changing tasks, and changing environmental conditions. Indeed, this ability to continuously accrue knowledge and use past experience to learn new tasks quickly in continual settings is one of the key properties of an intelligent system. For complex and high-dimensional problems, simply updating the model continually with standard learning algorithms such as gradient descent may result in slow adaptation. Meta-learning can provide a powerful tool to accelerate adaptation yet is conventionally studied in batch settings. In this paper, we study how metalearning can be applied to tackle online problems of this nature, simultaneously adapting to changing tasks and input distributions and meta-training the model in order to adapt more quickly in the future. Extending meta-learning into the online setting presents its own challenges, and although several prior methods have studied related problems, they generally require a discrete notion of tasks, with known ground-truth task boundaries. Such methods typically adapt to each task in sequence, resetting the model between tasks, rather than adapting continuously across tasks. In many real-world settings, such discrete boundaries are unavailable, and may not even exist. To address these settings, we propose a Fully Online Meta-Learning (FOML) algorithm, which does not require any ground truth knowledge about the task boundaries and stays fully online without resetting back to pre-trained weights. Our experiments show that FOML was able to learn new tasks faster than the state-of-the-art online learning methods on Rainbow-MNIST, CIFAR100 and CELEBA datasets.

show abstract

“…Continual-meta learning focuses on fast learning and remembering [25,35,33,41], often emphasising the online performance on OOD tasks [14]. As argued by Jerfel et al [41] modularity can be useful in this setting to minimize interference between tasks.…”

Section: Related Workmentioning

confidence: 99%

“…The goal of this setting is to construct a learner that quickly (i.e. within a few steps of gradient descent) learns tasks from new environments and relearns (or remembers) tasks from previously learned environments [41,14,34]. Methods applicable in this setting usually rely on gradient-based meta-learning strategies based on MAML [24].…”

Section: F Continual Meta-learningmentioning

confidence: 99%

Continual Learning via Local Module Composition

Ostapenko¹,

Rodríguez²,

Caccia³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Modularity is a compelling solution to continual learning (CL), the problem of modeling sequences of related tasks. Learning and then composing modules to solve different tasks provides an abstraction to address the principal challenges of CL including catastrophic forgetting, backward and forward transfer across tasks, and sub-linear model growth. We introduce local module composition (LMC), an approach to modular CL where each module is provided a local structural component that estimates a module's relevance to the input. Dynamic module composition is performed layer-wise based on local relevance scores. We demonstrate that agnosticity to task identities (IDs) arises from (local) structural learning that is module-specific as opposed to the task-and/or model-specific as in previous works, making LMC applicable to more CL settings compared to previous works. In addition, LMC also tracks statistics about the input distribution and adds new modules when outlier samples are detected. In the first set of experiments, LMC performs favorably compared to existing methods on the recent Continual Transfer-learning Benchmark without requiring task identities. In another study, we show that the locality of structural learning allows LMC to interpolate to related but unseen tasks (OOD), as well as to compose modular networks trained independently on different task sequences into a third modular network without any fine-tuning. Finally, in search for limitations of LMC we study it on more challenging sequences of 30 and 100 tasks, demonstrating that local module selection becomes much more challenging in presence of a large number of candidate modules. In this setting best performing LMC spawns much fewer modules compared to an oracle based baseline, however it reaches a lower overall accuracy. The codebase is available under https://github.com/oleksost/LMC.

show abstract

Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning

Cited by 10 publications

References 0 publications

Towards mental time travel: a hierarchical memory for reinforcement learning agents

Towards mental time travel: a hierarchical memory for reinforcement learning agents

Fully Online Meta-Learning Without Task Boundaries

Continual Learning via Local Module Composition

Contact Info

Product

Resources

About