Revisiting Distillation and Incremental Classifier Learning

Abstract

One of the key differences between the learning mechanism of humans and Artificial Neural Networks (ANNs) is the ability of humans to learn one task at a time. ANNs, on the other hand, can only learn multiple tasks simultaneously. Any attempts at learning new tasks incrementally cause them to completely forget about previous tasks. This lack of ability to learn incrementally, called Catastrophic Forgetting, is considered a major hurdle in building a true AI system. In this paper, our goal is to isolate the truly effective existing ideas for incremental learning from those that only work under certain conditions. To this end, we first thoroughly analyze the current state of the art (iCaRL) method for incremental learning and demonstrate that the good performance of the system is not because of the reasons presented in the existing literature. We conclude that the success of iCaRL is primarily due to knowledge distillation and recognize a key limitation of knowledge distillation, i.e, it often leads to bias in classifiers. Finally, we propose a dynamic threshold moving algorithm that is able to successfully remove this bias. We demonstrate the effectiveness of our algorithm on CIFAR100 and MNIST datasets showing near-optimal results. Our implementation is available at : https://github.com/Khurramjaved96/ incremental-learning.

Incremental learning for the detection and classification of GAN-generated images

“…Since none of these networks has been proposed in a classincremental scenario, to be as fair as possible we fine-tune them using all the data from the new classes and the limited exemplar set P available. We also compare our proposal with the basic iCaRL version (that is, Single Task and Single Classifier) and two more strategies proposed in [20], referred to as S-Classifier and GS-Classifier. Again, to ensure a fair comparison, we use XceptionNet as feature extractor for all these methods.…”

Efficient Incremental Learning Using Dynamic Correction Vector

“…Second, the Nearest-Mean-of-Exemplars (NME) classifier, rather than deep neural networks, is used to learn through back-propagation. Javed et al proposed a dynamic threshold moving (DTM) [4] algorithm. In general, threshold moving is often an effective tool for solving data imbalance problems.…”

“…To address these problems, researchers have proposed various solutions [4]- [7]. However, the problem of catastrophic forgetting persists.…”

“…In general, the existing incremental learning techniques can be categorized into four parts, which are 1) representative memory based [4], 2) context neuron gating based [9], 3) limiting weight updating direction and sizes [10], and 4) training a specific network for each task [11]. However, the performance of those techniques are limited.…”

Single-Net Continual Learning with Progressive Segmented Training

“…In the family of the single-network methods, previous works have explored the regularization methods [1,4,14,18,33], the parameter isolation methods [21,22] and the memory rehearsal methods [3,5,12,20,27]. The regularization methods leverage a penalty term in the loss function to regularize the parameters when updating for new tasks.…”