Maintaining Discrimination and Fairness in Class Incremental Learning

Zhao, Bowen; Xiao, Xi; Gan, Guojun; Zhang, Bin; Xia, Shu-Tao

doi:10.1109/cvpr42600.2020.01322

Cited by 294 publications

(206 citation statements)

References 13 publications

Supporting

Mentioning

186

Contrasting

Unclassified

Order By: Relevance

“…Xiang et al [ 78 ] proposed an algorithm based on dynamic correction vectors to solve the deviation from knowledge distillation and model overfitting problems. Zhao et al [ 79 ] combined weight adjustment and knowledge distillation in order to balance the new and old knowledge. Javed et al [ 80 ] proposed a dynamic threshold shift method to improve the limitations of the deviation in a general knowledge distillation model.…”

Section: Methods Descriptionmentioning

confidence: 99%

An Appraisal of Incremental Learning Methods

Luo

Yin

Bai

et al. 2020

Entropy

View full text Add to dashboard Cite

As a special case of machine learning, incremental learning can acquire useful knowledge from incoming data continuously while it does not need to access the original data. It is expected to have the ability of memorization and it is regarded as one of the ultimate goals of artificial intelligence technology. However, incremental learning remains a long term challenge. Modern deep neural network models achieve outstanding performance on stationary data distributions with batch training. This restriction leads to catastrophic forgetting for incremental learning scenarios since the distribution of incoming data is unknown and has a highly different probability from the old data. Therefore, a model must be both plastic to acquire new knowledge and stable to consolidate existing knowledge. This review aims to draw a systematic review of the state of the art of incremental learning methods. Published reports are selected from Web of Science, IEEEXplore, and DBLP databases up to May 2020. Each paper is reviewed according to the types: architectural strategy, regularization strategy and rehearsal and pseudo-rehearsal strategy. We compare and discuss different methods. Moreover, the development trend and research focus are given. It is concluded that incremental learning is still a hot research area and will be for a long period. More attention should be paid to the exploration of both biological systems and computational models.

show abstract

Section: Methods Descriptionmentioning

confidence: 99%

An Appraisal of Incremental Learning Methods

Luo

Yin

Bai

et al. 2020

Entropy

View full text Add to dashboard Cite

show abstract

“…Rehearsal / replay consists in replaying old data to the model at each new training step. One way is to select some samples from the incoming data stream and store them inside an episodic memory [4], [19], [22], [26]. Rehearsal is currently the most successful strategy to counter forgetting.…”

Section: A Class-incremental Learningmentioning

confidence: 99%

“…Regularization is a strategy consisting in implementing a protectionist policy on learned knowledge. Usually, the regularization is directly applied on the output layer using knowledge distillation [16], [19], [26]: the previous state of the model is used as a teacher for the new model in order to maintain the discrimination between old classes when optimizing new outputs. Other methods add the regularization on every weights of the model [1], [12], [24].…”

Section: A Class-incremental Learningmentioning

confidence: 99%

Semi-Supervised Class Incremental Learning

Lechat

Herbin

Jurie

2021

2020 25th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

This paper makes a contribution to the problem of incremental class learning, the principle of which is to sequentially introduce batches of samples annotated with new classes during the learning phase. The main objective is to reduce the drop in classification performance on old classes, a phenomenon commonly called catastrophic forgetting. We propose in this paper a new method which exploits the availability of a large quantity of non-annotated images in addition to the annotated batches. These images are used to regularize the classifier and give the feature space a more stable structure. We demonstrate on two image data sets, MNIST and STL-10, that our approach is able to improve the global performance of classifiers learned using an incremental learning protocol, even with annotated batches of small size.

show abstract

“…Regularization methods add specific regularization terms to consolidate knowledge learned before. Li and Hoiem (2017) introduced the knowledge distillation (Hinton et al, 2015) to penalize model logit change, and it has been widely employed in Rebuffi et al (2017);Castro et al (2018); ; Hou et al (2019); Zhao et al (2019). Another direction is to regularize parameters crucial to old knowledge according to various importance measures (Kirkpatrick et al, 2017;Zenke et al, 2017;Aljundi et al, 2018).…”

Section: Related Workmentioning

confidence: 99%

“…Exemplar replay methods store past samples, a.k.a exemplars, and replay them periodically. Instead of selecting exemplars at random, Rebuffi et al (2017) incorporated the Herding technique (Welling, 2009) to choose exemplars that best approximate the mean feature vector of a class, and it is widely used in Castro et al (2018); ; Hou et al (2019); Zhao et al (2019); Mi et al (2020a,b). Ramalho and Garnelo (2019) proposed to store samples that the model is least confident.…”

Section: Related Workmentioning

confidence: 99%

Continual Learning for Natural Language Generation in Task-oriented Dialog Systems

Mi¹,

Chen²,

Zhao³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Natural language generation (NLG) is an essential component of task-oriented dialog systems. Despite the recent success of neural approaches for NLG, they are typically developed in an offline manner for particular domains. To better fit real-life applications where new data come in a stream, we study NLG in a "continual learning" setting to expand its knowledge to new domains or functionalities incrementally. The major challenge towards this goal is catastrophic forgetting, meaning that a continually trained model tends to forget the knowledge it has learned before. To this end, we propose a method called ARPER (Adaptively Regularized Prioritized Exemplar Replay) by replaying prioritized historical exemplars, together with an adaptive regularization technique based on Elastic Weight Consolidation. Extensive experiments to continually learn new domains and intents are conducted on MultiWoZ-2.0 to benchmark ARPER with a wide range of techniques. Empirical results demonstrate that ARPER significantly outperforms other methods by effectively mitigating the detrimental catastrophic forgetting issue.

show abstract

Maintaining Discrimination and Fairness in Class Incremental Learning

Cited by 294 publications

References 13 publications

An Appraisal of Incremental Learning Methods

An Appraisal of Incremental Learning Methods

Semi-Supervised Class Incremental Learning

Continual Learning for Natural Language Generation in Task-oriented Dialog Systems

Contact Info

Product

Resources

About