Continual Unsupervised Representation Learning

Rao, Dushyant; Visin, Francesco; Rusu, Andrei A.; Teh, Yee Whye; Pascanu, Razvan; Hadsell, Raia

doi:10.48550/arxiv.1910.14481

Cited by 10 publications

(16 citation statements)

References 25 publications

(41 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use a Bit-Swap with 8 hierarchical latent variables with a Markov chain structure for each level of codes. In our preliminary experiments, we find the Resnet-32 used in previous replay-based methods [26,36,39] causes underfitting of our DRR in some cases. We use a ResNet-18 [13] used in [25] as a classifier (see Appendix for further discussion).…”

Section: Methodsmentioning

confidence: 89%

“…For DRR and IB-DRR * we only save codes for seen classes. For IB-DRR, we follow [26,36,39] and save additionally 20 raw exemplars per class for seen classes.…”

Section: Methodsmentioning

confidence: 99%

“…There were many variants of Resnet with a different number of layers and different 'inplanes'. In recent incremental learning works researchers prefer a Resnet-32 [26,39,37] with 'in-planes = 16' that had more layers but lower capacity than original Resnet-18 with 'in-planes = 64'. In our preliminary experiments, we found that Resnet-32 with 'in-planes = 16' resulted in underfitting of our models in some cases, e.g., if a strong Data Augmentation was applied.…”

Section: The Choice Of Resnet For Cifar-100mentioning

confidence: 99%

See 2 more Smart Citations

IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay

Jian¹,

Cetin²,

Çeliktutan³

2021

Preprint

View full text Add to dashboard Cite

Incremental learning aims to enable machine learning models to continuously acquire new knowledge given new classes, while maintaining the knowledge already learned for old classes. Saving a subset of training samples of previously seen classes in the memory and replaying them during new training phases is proven to be an efficient and effective way to fulfil this aim. It is evident that the larger number of exemplars the model inherits the better performance it can achieve. However, finding a trade-off between the model performance and the number of samples to save for each class is still an open problem for replay-based incremental learning and is increasingly desirable for real-life applications. In this paper, we approach this open problem by tapping into a two-step compression approach. The first step is a lossy compression, we propose to encode input images and save their discrete latent representations in the form of 'codes' that are learned using a hierarchical Vector Quantised Variational Autoencoder (VQ-VAE). In the second step, we further compress 'codes' losslessly by learning a hierarchical latent variable model with bitsback asymmetric numeral systems (BB-ANS). To compensate for the information lost in the first step compression, we introduce an Information Back (IB) mechanism that utilizes raw exemplars for a contrastive learning loss to regularise the training of a classifier. By maintaining all seen exemplars' representations in the format of 'codes', Discrete Representation Replay (DRR) outperforms the stateof-art method on CIFAR-100 by a margin of 4% average accuracy with a much less memory cost required for saving samples. Incorporated with IB and saving a small set of old raw exemplars as well, the average accuracy of DRR can be further improved by 2%.

show abstract

Section: Methodsmentioning

confidence: 89%

“…For DRR and IB-DRR * we only save codes for seen classes. For IB-DRR, we follow [26,36,39] and save additionally 20 raw exemplars per class for seen classes.…”

Section: Methodsmentioning

confidence: 99%

Section: The Choice Of Resnet For Cifar-100mentioning

confidence: 99%

See 1 more Smart Citation

IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay

Jian¹,

Cetin²,

Çeliktutan³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Lifelong (or continual) learning describes the scenario in which new tasks arrive sequentially and should be incorporated into the current model, retaining previous knowledge (Parisi et al, 2019). Approaches to lifelong learning are mainly aimed to mitigate catastrophic forgetting (Rao et al, 2019;Ramapuram et al, 2020;Ye and Bors, 2020). According to Parisi et al (2019), there are three main approaches to lifelong learning: (i) retraining with regularisation; (ii) network expansion; (iii) selective network retraining and expansion.…”

Section: Related Workmentioning

confidence: 99%

Transfer Learning via Test-time Neural Networks Aggregation

Casella

Chisari

Battiato

et al. 2022

Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Application

View full text Add to dashboard Cite

It has been demonstrated that deep neural networks outperform traditional machine learning. However, deep networks lack generalisability, that is, they will not perform as good as in a new (testing) set drawn from a different distribution due to the domain shift. In order to tackle this known issue, several transfer learning approaches have been proposed, where the knowledge of a trained model is transferred into another to improve performance with different data. However, most of these approaches require additional training steps, or they suffer from catastrophic forgetting that occurs when a trained model has overwritten previously learnt knowledge. We address both problems with a novel transfer learning approach that uses network aggregation. We train dataset-specific networks together with an aggregation network in a unified framework. The loss function includes two main components: a task-specific loss (such as cross-entropy) and an aggregation loss. The proposed aggregation loss allows our model to learn how trained deep network parameters can be aggregated with an aggregation operator. We demonstrate that the proposed approach learns model aggregation at test time without any further training step, reducing the burden of transfer learning to a simple arithmetical operation. The proposed approach achieves comparable performance w.r.t. the baseline. Besides, if the aggregation operator has an inverse, we will show that our model also inherently allows for selective forgetting, i.e., the aggregated model can forget one of the datasets it was trained on, retaining information on the others.

show abstract

“…Although large-scale datasets can cover many scenarios, when new classes data appear, the supervised model still cannot fit them well. Some works try to improve the supervised CIL by using unsupervised methods [7,25]. However, they still base on supervised labels, and the gap between them and iCaRL is tiny.…”

Section: Self-supervised Learningmentioning

confidence: 99%

Self-Supervised Class Incremental Learning

Ni¹,

Tang²,

Zhuang³

2021

Preprint

View full text Add to dashboard Cite

Existing Class Incremental Learning (CIL) methods are based on a supervised classification framework sensitive to data labels. When updating them based on the new class data, they suffer from catastrophic forgetting: the model cannot discern old class data clearly from the new. In this paper, we explore the performance of Self-Supervised representation learning in Class Incremental Learning (SS-CIL) for the first time, which discards data labels and the model's classifiers. To comprehensively discuss the difference in performance between supervised and selfsupervised methods in CIL, we set up three different class incremental schemes: Random Class Scheme, Semantic Class Scheme, and Cluster Scheme, to simulate various class incremental learning scenarios. Besides, we propose Linear Evaluation Protocol (LEP) and Generalization Evaluation Protocol (GEP) to metric the model's representation classification ability and generalization in CIL. Our experiments (on ImageNet-100 and ImageNet) show that SSCIL has better anti-forgetting ability and robustness than supervised strategies in CIL.To understand what alleviates the catastrophic forgetting in SSCIL, we study the major components of SSCIL and conclude that (1) the composition of different data augmentation improves the quality of the model's representation and the Grayscale operation reduces the system noise of data augmentation in SSCIL. (2) the projector, like a buffer, reduces unnecessary parameter updates of the model in SS-CIL and increases the robustness of the model. Although the performance of SSCIL is significantly higher than supervised methods in CIL, there is still an apparent gap with joint learning. Our exploration gives a baseline of self-supervised class incremental learning on large-scale datasets and contributes some forward strategies for mitigating the catastrophic forgetting in CIL.

show abstract

Continual Unsupervised Representation Learning

Cited by 10 publications

References 25 publications

IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay

IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay

Transfer Learning via Test-time Neural Networks Aggregation

Self-Supervised Class Incremental Learning

Contact Info

Product

Resources

About