IL2M: Class Incremental Learning With Dual Memory

Abstract

This paper presents a class incremental learning (IL) method which exploits fine tuning and a dual memory to reduce the negative effect of catastrophic forgetting in image recognition. First, we simplify the current fine tuning based approaches which use a combination of classification and distillation losses to compensate for the limited availability of past data. We find that the distillation term actually hurts performance when a memory is allowed. Then, we modify the usual class IL memory component. Similar to existing works, a first memory stores exemplar images of past classes. A second memory is introduced here to store past class statistics obtained when they were initially learned. The intuition here is that classes are best modeled when all their data are available and that their initial statistics are useful across different incremental states. A prediction bias towards newly learned classes appears during inference because the dataset is imbalanced in their favor. The challenge is to make predictions of new and past classes more comparable. To do this, scores of past classes are rectified by leveraging contents from both memories. The method has negligible added cost, both in terms of memory and of inference complexity. Experiments with three large public datasets show that the proposed approach is more effective than a range of competitive state-of-the-art methods.

Learning Then, Learning Now, and Every Second in Between: Lifelong Learning With a Simulated Humanoid Robot

“…The architecture adaptation is data-driven by learning which neurons need to be trained and what is the maximum adaptation applied to these neurons using variational inference. The Incremental Learning With Dual Memory (IL2M) (Belouadah and Popescu, 2019 ) consists of two parts: (1) a deep learning model that is incrementally trained on new samples and a constant number of previous representative samples, and (2) an additional memory that stores previous task statistics which are periodically used to rectify the network. The Growing Dual Memory (GDM) approach of Parisi et al ( 2018 ) is an architectural and a generative replay approach.…”

IExpressNet

“…However, when there is a strong correlation between the additional classes, the results of this methods are far from ideal. Class Incremental Learning With Dual Memory (IL2M) [5] propose to construct a training set including exemplar and new class data to realize fine-tuning of previous networks. In test time, the results of classification are generated by rectifying the prediction scores of past classes based on some statistics.…”

“…Followed by benchmark protocol in other class incremental methods [5,53], the classes are arranged in a fixed order. The first two classes are selected to train an initial network at the beginning.…”

“…As introduced in Related Work, the third deep class incremental learning method presents most outstanding performance. Thus, we select LwF [36], IL2M [5], iCaRL, and iCaRL-h1 [53] as the baselines to compare with our approach. Different from iCaRL, iCaRL-h1 use FC layer as the final classifier instead of NEM classifier.…”

“…Fine-tuning. Fine-tuning(FT) [5] aims at adding new dimensions at FC layer for new expressions and tuning the whole network on the training samples of new expressions. Traditional Approach.…”