Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning

Smith, James Seale; Hsu, Yen-Chang; Balloch, Jonathan; Shen, Yilin; Jin, Hongxia; Kira, Zsolt

doi:10.1109/iccv48922.2021.00924

Cited by 89 publications

(32 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Avoiding privacy concerns, this work also follows a line of work that doesn't store real examples for experience replay, such as generating examples by GAN (Atkinson et al, 2018), synthesizing examples (Xu et al, 2022) by model-inversion (Smith et al, 2021b), and using unlabeled data in the learning environment (Smith et al, 2021a). In language domain, LAMOL (Sun et al, 2019) trains the language model to solve current tasks and generate current training examples simultaneously, then this model can generate "pseudo" old examples for replay before any new tasks.…”

Section: Related Workmentioning

confidence: 99%

Continual Sequence Generation with Adaptive Compositional Modules

Zhang¹,

Wang²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

Continual learning is essential for real-world deployment when there is a need to quickly adapt the model to new tasks without forgetting knowledge of old tasks. Existing work on continual sequence generation either always reuses existing parameters to learn new tasks, which is vulnerable to catastrophic forgetting on dissimilar tasks, or blindly adds new parameters for every new task, which could prevent knowledge sharing between similar tasks. To get the best of both worlds, in this work, we propose continual sequence generation with adaptive compositional modules to adaptively add modules in transformer architectures and compose both old and new modules for new tasks. We also incorporate pseudo experience replay to facilitate knowledge transfer in those shared modules. Experiment results on various sequences of generation tasks show that our framework can adaptively add modules or reuse modules based on task similarity, outperforming state-of-the-art baselines in terms of both performance and parameter efficiency. We make our code public at https://github.com/GT-SALT/ Adaptive-Compositional-Modules.

show abstract

Section: Related Workmentioning

confidence: 99%

Continual Sequence Generation with Adaptive Compositional Modules

Zhang¹,

Wang²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Continual learning focuses on alleviating catastrophic forgetting while being discriminative for newly learned classes. To solve this problem, many work [5,6,12,48,78] propose to review knowledge by rehearsal-based mechanism. The knowledge can be stored by multiple types, like examples [5,7,10,12,74,84], prototypes [36,107,108], generative networks [61], etc.…”

Section: Related Workmentioning

confidence: 99%

Representation Compensation Networks for Continual Semantic Segmentation

Zhang¹,

Xiao²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this work, we study the continual semantic segmentation problem, where the deep neural networks are required to incorporate new classes continually without catastrophic forgetting. We propose to use a structural re-parameterization mechanism, named representation compensation (RC) module, to decouple the representation learning of both old and new knowledge. The RC module consists of two dynamically evolved branches with one frozen and one trainable. Besides, we design a pooled cube knowledge distillation strategy on both spatial and channel dimensions to further enhance the plasticity and stability of the model. We conduct experiments on two challenging continual semantic segmentation scenarios, continual class segmentation and continual domain segmentation. Without any extra computational overhead and parameters during inference, our method outperforms state-of-the-art performance. The code is available at https://github.com/zhangchbin/RCIL.

show abstract

“…In comparison, distillation-based approaches can be directly applied to continual learning of new classes by distilling knowledge from the old classifier (for old classes) to the new classifier (for both new and old classes) during learning new knowledge [18,19,37,38,39,40,41,42,43], where the old knowledge is often implicitly represented by soft outputs of old classifier with stored small amount of old images and/or new classes of images as the inputs. A distillation loss is added to the original cross-entropy loss during training the new classifier, where the distillation loss helps the new classifier have similar relevant output compared to the output of the old classifier for any input image.…”

Section: Related Workmentioning

confidence: 99%

Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractor

Yang¹,

Cui²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning has shown its human-level performance in various applications. However, current deep learning models are characterised by catastrophic forgetting of old knowledge when learning new classes. This poses a challenge particularly in intelligent diagnosis systems where initially only training data of a limited number of diseases are available. In this case, updating the intelligent system with data of new diseases would inevitably downgrade its performance on previously learned diseases. Inspired by the process of learning new knowledge in human brains, we propose a Bayesian generative model for continual learning built on a fixed pre-trained feature extractor. In this model, knowledge of each old class can be compactly represented by a collection of statistical distributions, e.g. with Gaussian mixture models, and naturally kept from

show abstract

Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning

Cited by 89 publications

References 26 publications

Continual Sequence Generation with Adaptive Compositional Modules

Continual Sequence Generation with Adaptive Compositional Modules

Representation Compensation Networks for Continual Semantic Segmentation

Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractor

Contact Info

Product

Resources

About