Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches

Lomonaco, Vincenzo; Maltoni, Davide; Pellegrini, Lorenzo

doi:10.1109/cvprw50498.2020.00131

Cited by 41 publications

(38 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In general, this approach shows satisfactory performance in problems that involve few tasks, however, when the number of tasks increases, problems such as accumulated drifting in weight values and interference among them, make this approach difficult to scale. Alternatives to decrease the interference between tasks are presented in [35], [39], here the authors propose to completely freeze previously trained weights eliminating interference but inhibiting information transfer between tasks.…”

Section: A Catastrophic Forgettingmentioning

confidence: 99%

Overcoming Catastrophic Forgetting Using Sparse Coding and Meta Learning

2021

View full text Add to dashboard Cite

Continuous learning occurs naturally in human beings. However, Deep Learning methods suffer from a problem known as Catastrophic Forgetting (CF) that consists of a model drastically decreasing its performance on previously learned tasks when it is sequentially trained on new tasks. This situation, known as task interference, occurs when a network modifies relevant weight values as it learns a new task. In this work, we propose two main strategies to face the problem of task interference in convolutional neural networks. First, we use a sparse coding technique to adaptively allocate model capacity to different tasks avoiding interference between them. Specifically, we use a strategy based on group sparse regularization to specialize groups of parameters to learn each task. Afterward, by adding binary masks, we can freeze these groups of parameters, using the rest of the network to learn new tasks. Second, we use a meta learning technique to foster knowledge transfer among tasks, encouraging weight reusability instead of overwriting. Specifically, we use an optimization strategy based on episodic training to foster learning weights that are expected to be useful to solve future tasks. Together, these two strategies help us to avoid interference by preserving compatibility with previous and future weight values. Using this approach, we achieve stateof-the-art results on popular benchmarks used to test techniques to avoid CF. In particular, we conduct an ablation study to identify the contribution of each component of the proposed method, demonstrating its ability to avoid retroactive interference with previous tasks and to promote knowledge transfer to future tasks.

show abstract

Section: A Catastrophic Forgettingmentioning

confidence: 99%

Overcoming Catastrophic Forgetting Using Sparse Coding and Meta Learning

2021

View full text Add to dashboard Cite

show abstract

“…Even though these scenarios propose alternative learning settings, none of them really address and discuss the assumptions and constraints that characterize the class-incremental scenario. In Section 3, we will present few notable exceptions (Stojanov et al, 2019;Lomonaco et al, 2020;Thai et al, 2021) that work on continual learning scenarios for classification based on repetition of previously seen classes. We now turn the attention to some of the limitations caused by the no repetition constraint of classincremental.…”

Section: Class-incremental Scenariosmentioning

confidence: 99%

“…Their scenario with repeated exposures allowed to bridge the gap with the joint training performances without using any explicit replay of previous patterns, but only by leveraging the natural replay occurring in the environment. The work done by Lomonaco et al (2020) proposes a flexible setup for CIR (see Figure 1 for a depiction of the resulting scenario). The authors based their experiments on the New Instances and Classes (NIC) continual learning scenario together with the CORe50 dataset, both introduced in Lomonaco and Maltoni (2017).…”

Section: Class-incremental With Repetition Scenariosmentioning

confidence: 99%

“…Therefore, it fits well the CIR family of scenarios. Due to the property of the NIC scenario, Lomonaco et al (2020) were able to experiment on a stream composed of a large number of experiences. The class-incremental counterpart produces a shorter stream since the total number of experiences is limited by the number of classes available in the dataset.…”

Section: Class-incremental With Repetition Scenariosmentioning

confidence: 99%

“…We now introduce a general protocol for build CIR data streams which, differently from the one by Lomonaco et al (2020), does not depend on the choice of a specific dataset. Instead, it can be instantiated on any dataset suitable for classification tasks.…”

Section: Class-incremental With Repetition Scenariosmentioning

confidence: 99%

See 2 more Smart Citations

Is Class-Incremental Enough for Continual Learning?

Cossu¹,

Graffieti²,

Pellegrini³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The ability of a model to learn continually can be empirically assessed in different continual learning scenarios. Each scenario defines the constraints and the opportunities of the learning environment. Here, we challenge the current trend in the continual learning literature to experiment mainly on class-incremental scenarios, where classes present in one experience are never revisited. We posit that an excessive focus on this setting may be limiting for future research on continual learning, since class-incremental scenarios artificially exacerbate catastrophic forgetting, at the expense of other important objectives like forward transfer and computational efficiency. In many real-world environments, in fact, repetition of previously encountered concepts occurs naturally and contributes to softening the disruption of previous knowledge. We advocate for a more in-depth study of alternative continual learning scenarios, in which repetition is integrated by design in the stream of incoming information. Starting from already existing proposals, we describe the advantages such class-incremental with repetition scenarios could offer for a more comprehensive assessment of continual learning models.

show abstract