Initial Classifier Weights Replay for Memoryless Class Incremental Learning

Belouadah, Eden; Popescu, Adrian; Kanellos, Ioannis

doi:10.48550/arxiv.2008.13710

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After the training of each new task, the classifier of each previous task is replaced by a scaled version of its initial stored classifier with the help of aggregate statistics. Another similar method [94] was also proposed to standardize the stored initial classifier of each task, resulting in a fair and balanced classification among different tasks. Chaudhry et al [92] proposed Hindsight Anchor Learning (HAL) to store an anchor per task in addition to data instances.…”

Section: It Uses Classification Uncertainty and Data Augmentation To ...mentioning

confidence: 99%

Recent Advances of Continual Learning in Computer Vision: An Overview

Qu,

Rahmani,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Section: It Uses Classification Uncertainty and Data Augmentation To ...mentioning

confidence: 99%

Recent Advances of Continual Learning in Computer Vision: An Overview

Qu,

Rahmani,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…SI [12] uses the path integral over the optimization trajectory. Recently, Belouadah et al [24], [25] assumed that all weights of the model should be normalized across tasks and standardized the initial classifier weights. Different from per-parameter regularization, LwF [11] adds regularization on top of the network output.…”

Section: Related Work a Class Incremental Learningmentioning

confidence: 99%

Self-distilled Knowledge Delegator for Exemplar-free Class Incremental Learning

Ye¹,

Li²,

Zhong³

et al. 2022

Preprint

View full text Add to dashboard Cite

Exemplar-free incremental learning is extremely challenging due to inaccessibility of data from old tasks. In this paper, we attempt to exploit the knowledge encoded in a previously trained classification model to handle the catastrophic forgetting problem in continual learning. Specifically, we introduce a so-called knowledge delegator, which is capable of transferring knowledge from the trained model to a randomly reinitialized new model by generating informative samples. Given the previous model only, the delegator is effectively learned using a self-distillation mechanism in a data-free manner. The knowledge extracted by the delegator is then utilized to maintain the performance of the model on old tasks in incremental learning. This simple incremental learning framework surpasses existing exemplar-free methods by a large margin on four widely used class incremental benchmarks, namely CIFAR-100, ImageNet-Subset, Caltech-101 and Flowers-102. Notably, we achieve comparable performance to some exemplar-based methods without accessing any exemplars.

show abstract

“…Therefore, there is a need for IL with a reasonable balance between accuracy, memory consumption, and training efficiency. In IoT, less or zero memory for historical data are preferred during the continuous evolving [11].…”

Section: Introductionmentioning

confidence: 99%

“…Other works require incrementally training task-related generative models for knowledge replay, but these generative models require notorious efforts [13]. In addition, there are several attempts to either use regularization or knowledge distillation to implement memoryless methods to prevent DNNs from forgetting [11]. Balancing between learning and forgetting is difficult, especially when the internal mechanism of catastrophic forgetting is not yet clear.…”

Section: Introductionmentioning

confidence: 99%

Class-Incremental Learning for Wireless Device Identification in IoT

Liu¹,

Wang²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep Learning (DL) has been utilized pervasively in the Internet of Things (IoT). One typical application of DL in IoT is device identification from wireless signals, namely Noncryptographic Device Identification (NDI). However, learning components in NDI systems have to evolve to adapt to operational variations, such a paradigm is termed as Incremental Learning (IL). Various IL algorithms have been proposed and many of them require dedicated space to store the increasing amount of historical data, and therefore, they are not suitable for IoT or mobile applications. However, conventional IL schemes can not provide satisfying performance when historical data are not available. In this paper, we address the IL problem in NDI from a new perspective, firstly, we provide a new metric to measure the degree of topological maturity of DNN models from the degree of conflict of class-specific fingerprints. We discover that an important cause for performance degradation in IL enabled NDI is owing to the conflict of devices' fingerprints. Second, we also show that the conventional IL schemes can lead to low topological maturity of DNN models in NDI systems. Thirdly, we propose a new Channel Separation Enabled Incremental Learning (CSIL) scheme without using historical data, in which our strategy can automatically separate devices' fingerprints in different learning stages and avoid potential conflict. Finally, We evaluated the effectiveness of the proposed framework using real data from ADS-B (Automatic Dependent Surveillance-Broadcast), an application of IoT in aviation. The proposed framework has the potential to be applied to accurate identification of IoT devices in a variety of IoT applications and services. Data and code available at IEEE Dataport

show abstract

Initial Classifier Weights Replay for Memoryless Class Incremental Learning

Cited by 3 publications

References 0 publications

Recent Advances of Continual Learning in Computer Vision: An Overview

Recent Advances of Continual Learning in Computer Vision: An Overview

Self-distilled Knowledge Delegator for Exemplar-free Class Incremental Learning

Class-Incremental Learning for Wireless Device Identification in IoT

Contact Info

Product

Resources

About