Exploring the Challenges Towards Lifelong Fact Learning

Elhoseiny, Mohamed; Babiloni, Fabio; Aljundi, Rahaf; Rohrbach, Marcus; Paluri, Manohar; Tuytelaars, Tinne

doi:10.1007/978-3-030-20876-9_5

Cited by 5 publications

(4 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance in a class-incremental setting, an extra head could be added to the network each time a new category label appears. Alternatively, a projection into an embedding space could be used, as in [7], avoiding the need for a growing network architecture. These are directions for future work.…”

Section: Discussionmentioning

confidence: 99%

Task-Free Continual Learning

Aljundi

Kelchtermans²,

Tuytelaars

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

243

177

View full text Add to dashboard Cite

Methods proposed in the literature towards continual deep learning typically operate in a task-based sequential learning setup. A sequence of tasks is learned, one at a time, with all data of current task available but not of previous or future tasks. Task boundaries and identities are known at all times. This setup, however, is rarely encountered in practical applications. Therefore we investigate how to transform continual learning to an online setup. We develop a system that keeps on learning over time in a streaming fashion, with data distributions gradually changing and without the notion of separate tasks. To this end, we build on the work on Memory Aware Synapses, and show how this method can be made online by providing a protocol to decide i) when to update the importance weights, ii) which data to use to update them, and iii) how to accumulate the importance weights at each update step. Experimental results show the validity of the approach in the context of two applications: (self-)supervised learning of a face recognition model by watching soap series and learning a robot to avoid collisions. * Rahaf Aljundi and Klaas Kelchtermans contributed equally to this work and are listed in alphabetical order.

show abstract

Section: Discussionmentioning

confidence: 99%

Task-Free Continual Learning

Aljundi

Kelchtermans²,

Tuytelaars

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

243

177

View full text Add to dashboard Cite

show abstract

“…They are chosen because they are more computationally efficient than SI (Zenke, Poole, and Ganguli 2017) and more memory efficient than IMM (Lee et al 2017). Additionally, some experiments such as (Elhoseiny et al 2018) show that MAS has better performance overall. Improved Memory-based parameter adaptation (MBPA++) Sparse experience replay and local adaption for LLL as proposed in (d' Autume et al 2019).…”

Section: Lamal γmentioning

confidence: 99%

LAMOL: LAnguage MOdeling for Lifelong Language Learning

Sun,

Ho,

Lee

2019

Preprint

View full text Add to dashboard Cite

Most research on lifelong learning (LLL) applies to images or games, but not language. Here, we introduce LAMAL, a simple yet effective method for LLL based on language modeling. LAMAL replays pseudo samples of previous tasks while requiring no extra memory or model capacity. To be specific, LAMAL is a language model learning to solve the task and generate training samples at the same time. At the beginning of training a new task, the model generates some pseudo samples of previous tasks to train alongside the data of the new task. The results show that LAMAL prevents catastrophic forgetting without any sign of intransigence and can solve up to five very different language tasks sequentially with only one model. Overall, LAMAL outperforms previous methods by a considerable margin and is only 2-3% worse than multitasking which is usually considered as the upper bound of LLL. Our source code is available at https://github.com/xxx.

show abstract

“…Shmelkov et al [45] and Liu et al [26] investigated incremental object detection while [24,50] learned image generation. Elhoseiny et al [16] examined continual fact learning by utilizing a visual-semantic embedding. Other works [1,20,54] focused on a reinforcement learning task.…”

Section: Related Workmentioning

confidence: 99%

The Surprising Positive Knowledge Transfer in Continual 3D Object Shape Reconstruction

Thai¹,

Stojanov²,

Huang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Continual learning is known for suffering from catastrophic forgetting, a phenomenon where earlier learned concepts are forgotten at the expense of more recent samples. In this work, we challenge the assumption that continual learning is inevitably associated with catastrophic forgetting by presenting a set of tasks that surprisingly do not suffer from catastrophic forgetting when learned continually. We attempt to provide an insight into the property of these tasks that make them robust to catastrophic forgetting and the potential of having a proxy representation learning task for continual classification. We further introduce a novel yet simple algorithm, YASS that outperforms state-of-the-art methods in the class-incremental categorization learning task. Finally, we present DyRT, a novel tool for tracking the dynamics of representation learning in continual models. The codebase, dataset and pretrained models released with this article can be found at https://github.com/ngailapdi/CLRec.

show abstract

Exploring the Challenges Towards Lifelong Fact Learning

Cited by 5 publications

References 31 publications

Task-Free Continual Learning

Task-Free Continual Learning

LAMOL: LAnguage MOdeling for Lifelong Language Learning

The Surprising Positive Knowledge Transfer in Continual 3D Object Shape Reconstruction

Contact Info

Product

Resources

About